Retrieval-augmented generation (RAG) grounds a language model's answers in your own documents instead of relying only on what it learned during training, which is how most production .NET assistants answer questions about internal policies, product catalogs or support history accurately. This guide is for developers who already know how to call a chat model and now need the whole pipeline: parsing and chunking documents, embedding and storing them, retrieving and reranking the right passages, assembling a grounded prompt with citations, and measuring whether the system actually works. It assumes the vector storage mechanics from Embeddings and Vector Databases in .NET and builds the rest of the pipeline around them.

What Is Retrieval-Augmented Generation?#

RAG answers a question in two stages instead of one. First, a retrieval step searches an external knowledge source, almost always a vector store, for the passages most relevant to the question. Second, a generation step hands those passages to a chat model as context and asks it to answer using them. The model still writes the answer, but the facts come from your data, not from parameters frozen at training time.

This matters for three practical reasons. Models cannot know about your private data or anything that changed after their training cutoff, so RAG is often the only way to get current, specific answers. Grounded answers can cite their sources, which lets a user verify a claim instead of trusting it blindly. Finally, updating a RAG system means re-indexing documents, which is minutes to hours of work, compared to retraining or fine-tuning a model, which is a much larger undertaking. The trade-off is a system with more moving parts: a chunking strategy, an index to keep fresh, and a retrieval step that can fail quietly by returning the wrong passages.

How RAG Works: The Pipeline#

A RAG system has two pipelines that run on different schedules. The ingestion pipeline runs whenever documents are added or changed: parse each source into text, split it into chunks, attach metadata, embed each chunk and upsert it into a vector store. The query pipeline runs on every user request: embed the question, search for candidate chunks, optionally rerank and filter them, assemble a prompt that includes the surviving chunks as context, and generate an answer that ideally cites where each fact came from.

Treat these as separate concerns with separate performance budgets. Ingestion can be slow and asynchronous, since a document being ten minutes stale rarely matters. Query-time retrieval has to be fast, typically well under a second, because it sits directly in the user's request path before the model even starts generating.

Getting Started: A Minimal RAG Query#

Once documents are indexed (the next sections cover how), a basic RAG query is a search followed by a chat call. This example uses the Microsoft.Extensions.VectorData and Microsoft.Extensions.AI abstractions described in the vector search guide:

C#
using Microsoft.Extensions.AI;
using Microsoft.Extensions.VectorData;

VectorStoreCollection<Guid, DocChunk> chunks = store.GetCollection<Guid, DocChunk>("doc_chunks");
string question = "What is our refund window for enterprise customers?";

List<DocChunk> hits = [];
await foreach (var result in chunks.SearchAsync(question, top: 5))
{
    hits.Add(result.Record);
}

string context = string.Join("\n\n", hits.Select((c, i) => $"[{i + 1}] {c.Text}"));
string prompt = $"""
    Answer using only the numbered context below. Cite sources like [1].
    If the answer is not in the context, say you don't know.

    Context:
    {context}

    Question: {question}
    """;

ChatResponse answer = await chatClient.GetResponseAsync(prompt);
Console.WriteLine(answer.Text);

Every later section refines one part of this: how hits are produced well, how the prompt is assembled safely, and how you know the answer was actually grounded.

Document Ingestion: Parsing, Chunking and Metadata#

Ingestion quality has more influence on answer quality than the choice of vector database. Three decisions matter most.

Parsing turns a source file into plain text you can chunk. Markdown and HTML need only light cleanup. PDFs and Office documents need a parser that preserves structure, such as headings and tables, because that structure drives good chunk boundaries; scanned or image-heavy PDFs need OCR. Multimodal AI in .NET: Vision, Audio and Speech covers document understanding services for that harder case. Preserve heading hierarchy during parsing even if you discard it before embedding, because it produces much better chunk boundaries than splitting on raw character counts.

Chunking balances two competing pressures: chunks must be small enough that each one is topically coherent and fits comfortably inside the model's context window alongside several other chunks, and large enough to retain the surrounding context a sentence needs to make sense on its own. A practical default is to split on headings and paragraphs first, then further split any paragraph that is still too large, using an overlap of 10 to 15 percent so a fact near a boundary doesn't get orphaned from its context. Measuring size in tokens, not characters, keeps chunks consistent with the model's actual limits:

C#
using Microsoft.ML.Tokenizers;

// TiktokenTokenizer.CreateForModel maps a model name to its tokenizer; cache the instance.
Tokenizer tokenizer = TiktokenTokenizer.CreateForModel("gpt-5");

IEnumerable<string> ChunkByHeadingThenSize(string markdown, int maxTokens, int overlapTokens)
{
    foreach (string section in SplitOnHeadings(markdown)) // your own heading-aware splitter
    {
        if (tokenizer.CountTokens(section) <= maxTokens)
        {
            yield return section;
            continue;
        }

        int start = 0;
        while (start < section.Length)
        {
            int end = tokenizer.GetIndexByTokenCount(section[start..], maxTokens, out _, out _);
            yield return section[start..(start + end)];

            if (start + end >= section.Length) break;
            start += Math.Max(end - overlapTokens, 1); // step back to create the overlap
        }
    }
}

Metadata turns a bag of chunks into something you can filter and audit. At minimum, store the source URI, a section heading or title, a chunk index for ordering, a last-modified timestamp for freshness, and any tenant or access-control fields your application needs to enforce at query time. Metadata design is a security boundary as much as a convenience: filtering on an indexed tenant field, as shown in the vector search guide, is what stops one customer's documents from surfacing in another customer's answers.

Embeddings and Vector Store Upserts#

Ingestion ends by embedding each chunk and upserting it, along with its metadata, into your vector store. This is a thin wrapper around the IEmbeddingGenerator and VectorStoreCollection types covered in Microsoft.Extensions.AI and Embeddings and Vector Databases in .NET, so only the ingestion-specific shape is new here: batch the upserts, and record enough metadata to delete and re-embed a source cleanly when it changes.

C#
async Task IngestDocumentAsync(SourceDocument doc, VectorStoreCollection<Guid, DocChunk> chunks,
    CancellationToken ct)
{
    // Remove any chunks from a previous version of this document before inserting the new ones.
    await chunks.DeleteAsync(await FindExistingChunkIdsAsync(doc.Uri, ct), ct);

    var records = ChunkByHeadingThenSize(doc.MarkdownText, maxTokens: 400, overlapTokens: 50)
        .Select((text, i) => new DocChunk
        {
            Id = Guid.NewGuid(),
            TenantId = doc.TenantId,
            SourceUri = doc.Uri,
            ChunkIndex = i,
            Text = text,
        });

    await chunks.UpsertAsync(records, ct); // embeddings are generated automatically on upsert
}

Retrieval: Hybrid Search, Filters and Reranking#

Plain vector search, top-k nearest neighbors by embedding distance, is a good baseline but has a well-known weak spot: it blurs exact tokens such as product codes, error numbers and names, because embeddings represent meaning rather than characters. Hybrid search and metadata filters, both covered in the vector search guide, fix the recall problem. What that guide does not cover is what to do once you have a first batch of candidates: reranking.

Reranking is a second, more expensive pass that reorders a small candidate set using a model that looks at the query and each candidate together, rather than comparing precomputed vectors. It catches relevant passages that vector search under-ranked and pushes down ones that matched on vocabulary but not intent. A practical, dependency-light way to rerank in .NET is to ask a fast chat model to score a batch of candidates with structured output, then keep only the top few:

C#
public sealed record RelevanceScore(int ChunkIndex, int Score); // 0-10, higher is more relevant

async Task<List<DocChunk>> RerankAsync(string question, List<DocChunk> candidates,
    IChatClient fastClient, int keep, CancellationToken ct)
{
    string listing = string.Join("\n",
        candidates.Select((c, i) => $"{i}: {c.Text[..Math.Min(300, c.Text.Length)]}"));
    string prompt = $"""
        Question: {question}

        Score how relevant each numbered passage below is to answering the question, 0 to 10.
        Passages:
        {listing}
        """;

    var result = await fastClient.GetResponseAsync<List<RelevanceScore>>(prompt, cancellationToken: ct);

    return result.TryGetResult(out var scores)
        ? [.. scores.OrderByDescending(s => s.Score).Take(keep).Select(s => candidates[s.ChunkIndex])]
        : candidates.Take(keep).ToList(); // fail open: fall back to the original order
}

For higher-scale systems, a dedicated cross-encoder reranker or a managed semantic ranker (Azure AI Search offers one) is cheaper per call and usually more accurate than using a general chat model as the judge; the pattern above is a good starting point that needs no extra infrastructure. Whichever approach you use, retrieve a wider candidate set than you plan to keep, for example the top 20 to 30 by vector or hybrid search, and let reranking narrow it to the 3 to 8 chunks that actually go in the prompt.

Assembling the Prompt: Context, Citations and Grounding#

A grounded prompt does three things: it clearly separates instructions from retrieved content, it numbers each source so the model can cite it, and it gives the model explicit permission to say it does not know. Interpolating retrieved text directly into an instruction string, without a clear boundary, leaves the prompt open to injected instructions hiding inside a document; wrapping each chunk with its number and source, as below, keeps the model's attention on citing rather than obeying whatever the source text says.

C#
public sealed record Citation(int Number, string SourceUri, string ChunkText);

(string prompt, List<Citation> citations) BuildGroundedPrompt(string question, List<DocChunk> chunks)
{
    var citations = chunks.Select((c, i) => new Citation(i + 1, c.SourceUri, c.Text)).ToList();
    string context = string.Join("\n\n",
        citations.Select(c => $"[{c.Number}] (source: {c.SourceUri})\n{c.ChunkText}"));

    string prompt = $"""
        You are a support assistant. Answer only from the context below.
        Cite every claim with its source number in brackets, like [1].
        If the context does not contain the answer, say so explicitly instead of guessing.

        Context:
        {context}

        Question: {question}
        """;

    return (prompt, citations);
}

After generation, map the [1], [2] markers the model produced back to the Citation list to render clickable source links in your UI. This closes the loop: the user sees not just an answer, but exactly which passages produced it.

Answer Grounding and Reducing Hallucination#

Grounding is a prompt discipline backed up by verification, not a single setting. The instructions above, "answer only from the context" and "say you don't know", remove most ungrounded answers, but models still occasionally state something plausible that is not in the provided passages, especially when the retrieved context is thin or off-topic. Three techniques catch what the prompt alone does not: keep the model's temperature low for factual answers, since grounded extraction benefits far more from consistency than from creative variation; require citations on every sentence and treat an uncited factual claim as a signal to re-check the retrieval, not just the prompt; and run an automated groundedness check, covered next, so hallucinations are caught in testing and monitoring rather than by an unhappy user.

Evaluating a RAG Pipeline: Relevance, Groundedness and Retrieval Quality#

A RAG pipeline has two places to measure separately: did retrieval find the right passages, and did generation use them faithfully. The Microsoft.Extensions.AI.Evaluation.Quality package ships evaluators for both: RetrievalEvaluator scores how well the retrieved chunks serve the query, GroundednessEvaluator scores how well the answer aligns with the supplied context, and RelevanceEvaluator scores how well the answer addresses the question itself. All three use an LLM as the judge and return a numeric score with a rationale.

C#
using Microsoft.Extensions.AI;
using Microsoft.Extensions.AI.Evaluation;
using Microsoft.Extensions.AI.Evaluation.Quality;

ChatConfiguration judge = new(judgeChatClient); // typically a strong, separate model from production

var groundedness = new GroundednessEvaluator();
EvaluationResult result = await groundedness.EvaluateAsync(
    messages: [new ChatMessage(ChatRole.User, question)],
    modelResponse: new ChatResponse(new ChatMessage(ChatRole.Assistant, answerText)),
    chatConfiguration: judge,
    additionalContext: [new GroundednessEvaluatorContext(retrievedContext: context)]);

NumericMetric groundednessScore = result.Get<NumericMetric>(GroundednessEvaluator.GroundednessMetricName);

Run these evaluators over a small, curated set of representative questions with known-good answers whenever you change chunking, the retrieval top count, the reranker, or the prompt, and again as a scheduled check against production traffic samples. Evaluating AI Applications in .NET covers the wider evaluation library in depth, including golden datasets, CI gates and cost control; this section is the RAG-specific slice of that larger picture.

Reference Architecture: ASP.NET Core with Background Ingestion#

A production RAG service separates the two pipelines described earlier into a background worker and a request-handling endpoint that share the same vector store and embedding generator through dependency injection.

C#
public sealed class IngestionWorker(
    IServiceScopeFactory scopeFactory, ILogger<IngestionWorker> logger) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        using PeriodicTimer timer = new(TimeSpan.FromMinutes(15));

        while (await timer.WaitForNextTickAsync(stoppingToken))
        {
            await using AsyncServiceScope scope = scopeFactory.CreateAsyncScope();
            var source = scope.ServiceProvider.GetRequiredService<IDocumentSource>();
            var chunks = scope.ServiceProvider.GetRequiredService<VectorStoreCollection<Guid, DocChunk>>();

            await foreach (SourceDocument doc in source.GetChangedSinceLastRunAsync(stoppingToken))
            {
                try
                {
                    await IngestDocumentAsync(doc, chunks, stoppingToken);
                }
                catch (Exception ex)
                {
                    logger.LogError(ex, "Failed to ingest {Uri}", doc.Uri);
                }
            }
        }
    }
}

The query endpoint stays a thin composition of the pieces built above: search, rerank, assemble, generate.

C#
app.MapPost("/support/ask", async (
    AskRequest request, VectorStoreCollection<Guid, DocChunk> chunks,
    IChatClient chatClient, IChatClient rerankClient, CancellationToken ct) =>
{
    List<DocChunk> candidates = [];
    await foreach (var r in chunks.SearchAsync(request.Question, top: 20,
        new() { Filter = c => c.TenantId == request.TenantId }, ct))
    {
        candidates.Add(r.Record);
    }

    List<DocChunk> best = await RerankAsync(request.Question, candidates, rerankClient, keep: 5, ct);
    (string prompt, List<Citation> citations) = BuildGroundedPrompt(request.Question, best);

    ChatResponse response = await chatClient.GetResponseAsync(prompt, cancellationToken: ct);
    return Results.Ok(new AskResponse(response.Text, citations));
});

public sealed record AskRequest(string TenantId, string Question);
public sealed record AskResponse(string Answer, List<Citation> Citations);

Register IngestionWorker with builder.Services.AddHostedService<IngestionWorker>(), and drive IDocumentSource from whatever holds your source content: a document store, a wiki API or a file share watcher. Keeping ingestion as a BackgroundService instead of running it inline means a slow parse or a flaky upstream source never blocks a user's question.

Common Failure Modes and How to Fix Them#

SymptomLikely causeFix
Answers ignore obviously relevant documentsChunks too large, diluting the embedding; or top too smallShrink chunk size, raise top, add hybrid search
Correct passage retrieved, wrong answerNo grounding instruction, or citations not enforcedTighten the prompt; require a citation per claim; add groundedness evaluation
Retrieval works in testing, degrades in productionIndex goes stale as source documents changeTrack document versions; re-embed on change; monitor ingestion lag
Exact codes or names never matchPure vector search misses literal tokensAdd hybrid (keyword plus vector) search
One tenant sees another tenant's contentFiltering applied after retrieval, or not at allFilter on an indexed tenant field before or during the vector search, never in application code afterward
Latency spikes under loadReranking every request against a large candidate setCap candidate count before reranking; cache embeddings for repeated queries

Best Practices#

  • Chunk by structure first, size second. Split on headings and paragraphs before falling back to a token budget, and measure that budget in tokens, not characters.
  • Retrieve wide, keep narrow. Pull more candidates than you need and let reranking or filters cut them down, rather than relying on top alone for precision.
  • Treat metadata as a security boundary. Enforce tenant and permission filters at the vector store, not after results come back.
  • Make "I don't know" a valid, expected answer. A model that always answers confidently, even without support in the context, is the main source of RAG hallucinations.
  • Version your ingestion, not just your index. Track which source version produced which chunks so you can delete and re-embed cleanly.
  • Evaluate retrieval and generation separately. A low Relevance score with a high Retrieval score points at the prompt or the model, not the index.

Common Pitfalls#

  • Skipping reranking entirely. Raw top-k vector search is good enough for demos but frequently surfaces topically related, practically useless passages in production.
  • Embedding titles and body separately without joining them. A chunk's embedding should usually include its heading; searching body text alone loses context that disambiguates near-duplicate sections.
  • Unbounded prompt growth. Adding "just one more chunk" for safety raises cost and, past a point, lowers answer quality by diluting the model's attention.
  • No re-ingestion trigger. A RAG system with no scheduled or event-driven re-ingestion silently serves stale answers forever.
  • Testing only happy-path questions. Include questions with no good answer in the corpus in your evaluation set, specifically to check the system says so.

RAG vs Fine-Tuning vs Long-Context Prompting#

ApproachBest forUpdate costWeakness
RAGLarge, changing knowledge bases; answers that need citationsRe-index changed documents (minutes)Retrieval quality caps answer quality
Fine-tuningTeaching a style, format or narrow skill, not new factsRetrain on new examples (hours to days)Poor fit for frequently changing facts; can still hallucinate
Long-context promptingSmall, stable document sets that fit in the context windowNone; just resend the documentsCost and latency grow with every request; no source filtering

These are complementary, not exclusive: many production systems fine-tune a model's tone or tool-use behavior while still using RAG for facts, and use long-context prompting for a single document a user just uploaded to the current conversation.

Frequently Asked Questions#

How big should my chunks be?#

There is no universal number; start around 200 to 500 tokens with 10 to 15 percent overlap, and tune it against your own evaluation set. Shorter chunks improve precision but need a higher top to keep recall, while longer chunks carry more context per hit but dilute the embedding and cost more per chunk retrieved.

Do I need a reranker, or is vector search with a high top-k enough?#

For a prototype, a high top without reranking is often fine. In production, reranking a wider candidate set down to a handful of chunks consistently improves answer quality more than any other single change, because it lets you retrieve broadly for recall and then narrow precisely for the prompt.

How is RAG different from just increasing the model's context window?#

A large context window still has a cost and latency that grow with every token you send, and it has no way to search across a knowledge base larger than the window; you would have to select what to include yourself, which is retrieval by another name. RAG scales to knowledge bases far larger than any context window and lets you filter and cite sources.

What's the biggest cause of poor RAG answers in practice?#

Ingestion quality: bad chunk boundaries, missing metadata and stale indexes cause more failures than the choice of embedding model, vector store or even the chat model. Fix retrieval before tuning the prompt.

Can RAG eliminate hallucinations completely?#

No. It reduces them substantially by giving the model real content to draw from, but a model can still misstate or overgeneralize what a passage says. Combine strong grounding instructions, mandatory citations and ongoing groundedness evaluation; treat the combination as risk reduction, not a guarantee.

Summary#

  • RAG separates a slow ingestion pipeline (parse, chunk, embed, upsert) from a fast query pipeline (retrieve, rerank, assemble, generate), and each has its own performance budget.
  • Chunk by structure, measure size in tokens, and carry metadata that supports both filtering and auditing.
  • Retrieve a wide candidate set with hybrid search, then rerank down to the few chunks that go in the prompt.
  • Ground answers with explicit citations and permission to say "I don't know," and verify grounding with GroundednessEvaluator, RelevanceEvaluator and RetrievalEvaluator.
  • Most production failures trace back to ingestion and retrieval, not the model, so evaluate and fix those first.

Further Reading#