RAG and Vector Search Interview Questions

Q1 Design a production RAG pipeline end to end. What are the major stages, and where does each one typically fail?#

Short answer: Ingest and parse source content, chunk it, generate embeddings, store vectors alongside metadata, and at query time embed the question, retrieve candidate chunks (often with hybrid search and reranking), assemble a grounded prompt, and generate a cited answer — five to seven stages, each with its own failure mode: ingestion loses structure, chunking splits meaning apart, an unversioned embedding model silently corrupts similarity, retrieval returns near-duplicates, and generation can ignore what was retrieved.

Ingestion is the stage architects most often underestimate: parsing a PDF, an HTML page or a Word document into clean text loses table structure, headers and reading order unless you specifically preserve them, and a chunk that silently merges a table's header row with an unrelated paragraph corrupts retrieval in a way that stays invisible until someone asks about that exact table. Chunking determines what unit of text becomes retrievable at all, which is why it gets its own question below. Embedding turns each chunk into a vector using a model that must stay consistent for a given index — mixing vectors from two different embedding models in the same space silently corrupts similarity search, so re-embedding is a deliberate migration, not a background job run casually. Storage has to carry metadata alongside the vector — source document, section, timestamp, permission scope — because filtering happens on that metadata, not on the vector itself. At query time, the naive version embeds the question and runs a nearest-neighbor search, but production systems commonly add hybrid retrieval and reranking before assembling the final prompt, and generation still needs an explicit instruction to answer only from the provided context and say when it cannot, or it will fill gaps with fluent, ungrounded text.

public interface IRagPipeline
{
    Task<IReadOnlyList<RetrievedChunk>> RetrieveAsync(string query, CancellationToken ct);
    Task<ChatResponse> GenerateAsync(
        string query, IReadOnlyList<RetrievedChunk> context, CancellationToken ct);
}

What interviewers look for: an end-to-end mental model with a named failure mode per stage, not just "embed and search" — architects are expected to reason about where quality is lost, not only where the vector database sits.

Common mistakes: describing RAG as "search plus an LLM call" without mentioning ingestion fidelity or embedding-model versioning, both of which cause real production incidents.

Q2 How do you choose a chunking strategy, and what goes wrong with naive fixed-size chunking?#

Short answer: Fixed-size chunking with some overlap is a reasonable default only for unstructured prose; for anything with inherent structure — code, tables, Markdown, legal or technical documents — a structure-aware strategy that splits on natural boundaries and only applies size limits within those boundaries retrieves far better, because it keeps semantically related content together instead of splitting one idea across two chunks that then compete for the same ranked slot.

Naive fixed-size chunking fails in two opposite ways. Chunks that are too small lose context — a sentence that says "it also throws under contention" is useless in retrieval without the preceding sentence naming what "it" is — while chunks that are too large dilute the embedding, mixing several topics into one vector that then matches everything a little and nothing well, which surfaces as retrieval returning technically related but useless chunks. Overlap, commonly ten to twenty percent of chunk size, mitigates boundary splitting but doesn't fix it, since a definition-and-example pair can still land split with neither half self-contained. Structure-aware chunking — splitting Markdown on headings, code on function or class boundaries, and treating tables as atomic units with their header row repeated in every chunk derived from them — consistently outperforms fixed-size splitting on real document sets, at the cost of needing a parser per content type instead of one generic splitter. Chunk size also interacts with both the embedding model's effective range and the generation model's context budget, but the retrieval-quality case for tight, coherent chunks holds regardless of how large the downstream context window is.

What interviewers look for: treating chunking as a semantics problem rather than a token-counting problem — a strong answer names structure-aware chunking and explains why fixed-size splitting degrades retrieval, not just that an alternative exists.

Common mistakes: using one global chunk-size constant for an entire mixed corpus, and assuming overlap eliminates boundary-splitting problems instead of only reducing them.

Q3 How do you choose an embedding model, and what happens when you need to change it later?#

Short answer: Choose based on domain fit, dimensionality versus storage and latency cost, and the languages and content types you actually index — evaluate candidates against a labeled retrieval set from your own corpus rather than trusting a generic leaderboard — and plan for the fact that changing embedding models later means re-embedding the entire corpus, because vectors from two different models are not comparable and cannot share an index.

The trap is choosing an embedding model the way you'd choose a chat model, by general capability, when what actually matters is how well it separates your documents on your queries. A model tuned mostly on general web text can perform poorly on dense technical or legal vocabulary, and a smaller, domain-appropriate model frequently beats a larger general-purpose one on retrieval precision for a specialized corpus, at a fraction of the cost. Dimensionality is a real trade-off, not just a number: higher-dimensional embeddings tend to capture more nuance but cost more to store and search, and some providers offer a smaller, truncated variant of the same model at a meaningful storage and latency saving with a modest quality cost, which is worth benchmarking rather than assuming the largest option is needed. The consequence that catches teams off guard is migration: because similarity is only meaningful within one embedding space, swapping models is not a configuration change, it's a backfill — every existing chunk has to be re-embedded into a new collection, typically alongside the old one with a planned cutover — which needs to be budgeted and versioned like a schema migration, including a rollback path if the new model retrieves worse on some query classes than the old one did.

IEmbeddingGenerator<string, Embedding<float>> embeddingGenerator = ...;
GeneratedEmbeddings<Embedding<float>> embeddings =
    await embeddingGenerator.GenerateAsync(chunks.Select(c => c.Text));

What interviewers look for: treating the embedding model as a versioned, migratable dependency rather than a one-time setup choice — the detail that most separates "built a RAG demo" from "operated a RAG index."

Follow-up questions:

• How would you run two embedding models side by side during a migration without downtime?
• Would you ever mix embedding models within a single collection? Why not?

Q4 What is hybrid search, and why do you need it on top of pure vector similarity?#

Short answer: Hybrid search combines dense vector similarity with a sparse keyword or lexical method such as BM25, typically merged with a fusion technique like reciprocal rank fusion, because embeddings are excellent at semantic similarity but systematically weak at exact-match signals — identifiers, error codes, acronyms, negation — that keyword search handles natively.

Pure vector search fails in specific, predictable ways: ask for an exact error code or a specific API name, and an embedding model treats it as just another token cluster, frequently ranking a semantically similar but wrong chunk above the one containing the literal string the user typed. Negation is a related weak point — chunks describing a request that does not retry and one that does retry can embed close together because the surrounding vocabulary is nearly identical, which a lexical match on "not" or "never" helps disambiguate. Reciprocal rank fusion is the common way to combine the two ranked lists without calibrating two incompatible score scales: each result's fused score sums 1 / (k + rank) across both the vector and keyword rankings, so a document that ranks well in either method contributes strongly without the raw scores needing to be comparable. In practice, hybrid retrieval is closer to a default than an optimization for any production system with real query diversity — the incremental cost is a keyword index alongside the vector index, which most vector databases and search platforms support natively, so the real design decision is fusion weighting, not whether to do it at all.

static double ReciprocalRankFusionScore(int vectorRank, int keywordRank, int k = 60) =>
    1.0 / (k + vectorRank) + 1.0 / (k + keywordRank);

What interviewers look for: naming a concrete failure class, such as exact identifiers or negation, rather than a vague "hybrid is better," and knowing at least one fusion method by name.

Common mistakes: assuming a strong embedding model makes keyword search unnecessary, and naively averaging raw vector and keyword scores instead of using a rank-based fusion method.

Q5 What does a reranking stage add, and where does it fit in the pipeline?#

Short answer: Reranking takes a larger, cheaply retrieved candidate set — for example the top 50 chunks from vector and keyword search — and re-scores it with a more expensive, more accurate model, typically a cross-encoder that evaluates the query and each candidate together, to produce the final handful that actually go into the prompt, trading extra latency and cost for meaningfully better precision at the top of the list.

Vector and keyword retrieval both score the query and each document independently, which is fast enough to search millions of chunks but caps how precisely they can judge relevance, since the query and document never actually interact during scoring. A cross-encoder reranker scores the query and a candidate chunk together in a single pass, which is far more accurate at judging true relevance but far too slow to run over an entire corpus, hence the two-stage design: cast a wide net cheaply, then spend expensive scoring only on a short list. This matters disproportionately for RAG because the generation model is highly sensitive to what lands in position one or two of the context; a reranking pass that moves the genuinely best chunk from rank eight to rank one measurably improves answer quality even when the underlying retrieval set didn't change. The trade-off is latency, since a reranking call adds real time to every query, so the candidate-set size fed into it — commonly the top 20 to 100 — and the reranker's own latency need to be tuned against the feature's overall latency target, and reranking is usually the first thing cut when that budget is tight.

What interviewers look for: the bi-encoder-versus-cross-encoder distinction stated precisely, and recognizing reranking as a cost-and-latency-for-precision trade rather than a free quality upgrade.

Follow-up questions:

• How would you decide the candidate-set size fed into the reranker?
• What would you do if the reranker itself became the latency bottleneck?

Q6 How do you evaluate a RAG system, specifically groundedness, as a repeatable process rather than a one-off check?#

Short answer: Evaluate retrieval and generation separately — retrieval metrics like recall@k and mean reciprocal rank against a labeled query-to-relevant-chunk set, and generation metrics like groundedness, whether every claim in the answer is actually supported by the retrieved context, typically scored with an automated rubric or an LLM-as-judge — and run both as a scheduled pipeline against a maintained golden dataset, not as an ad hoc check before a demo.

Retrieval and generation fail independently, and conflating them hides which one to fix: a system can retrieve the right chunk and still generate an ungrounded answer because the model ignored the context, and it can retrieve the wrong chunk and still generate a fluent, confident, wrong answer, which is the more dangerous failure because it doesn't look broken. Recall@k, whether a relevant chunk appeared anywhere in the top k, and mean reciprocal rank, how high the first relevant chunk ranked, measure retrieval quality against a labeled set of realistic queries mapped to the chunks that should answer them; building that labeled set is ongoing work, not a one-time task, since both the corpus and the query distribution drift. Groundedness is usually measured by decomposing the generated answer into individual claims and checking each one against the retrieved context, either with a structured rubric or a judge prompt specifically designed for this decomposition — Microsoft.Extensions.AI.Evaluation packages this kind of scoring as reusable evaluators rather than requiring every team to write its own judge prompt from scratch. The process that holds up in production runs this evaluation on a schedule against a versioned golden set, tracks scores over time as a regression signal, and treats a drop in groundedness the way you'd treat a failing test suite, not something discovered only when a user complains.

What interviewers look for: the retrieval-versus-generation split stated explicitly, plus a repeatable pipeline mental model rather than a one-time manual review.

Common mistakes: judging only whether the answer "sounds right" without checking it against the retrieved context, and never revisiting the golden evaluation set as the corpus changes.

Q7 Source documents change constantly, and some get retracted. How do you keep a RAG index from serving stale or contradictory information?#

Short answer: Treat the vector index as a derived, rebuildable artifact driven by change events from the source of truth — a change feed, a webhook, or a scheduled diff — so an edit or deletion propagates as a targeted re-index of just the affected chunks, plus a way to immediately suppress a retracted document from retrieval even before the full re-index completes.

The naive approach, a periodic full re-index, is simple but creates a real staleness window — hours or a day where the index still serves an outdated or retracted document — which is unacceptable for pricing, policy or safety-relevant content. An event-driven design instead reacts to the specific change: an edit invalidates, re-chunks and re-embeds only the chunks derived from that document; a deletion removes its vectors immediately; and a retraction can be handled faster than a full re-embed by flipping a suppressed metadata flag the retrieval filter checks first, buying time for the slower re-index without ever serving the retracted content in between. Every chunk's metadata should carry enough provenance — source document ID and a version or last-modified timestamp — to support both this targeted invalidation and a freshness disclosure to the end user when it matters. The harder version of this problem is contradiction, not just staleness: two versions of a policy can both still be indexed briefly during a rollout, and retrieval can surface both, so the design needs either strict single-active-version enforcement per logical document or an explicit recency signal fed into ranking so the newer version wins ties.

What interviewers look for: an event-driven, targeted-invalidation design instead of "just re-index nightly," and recognizing staleness and contradiction as related but distinct problems.

Common mistakes: relying solely on a full periodic rebuild for anything time-sensitive, and forgetting that a deleted or retracted source document must be actively removed, not just left to age out.

Q8 How do you make sure users only retrieve chunks they're actually authorized to see?#

Short answer: Enforce authorization as a filter applied during retrieval — a metadata predicate evaluated by the vector store alongside the similarity search — never as a filter applied to results after retrieval, because post-filtering silently shrinks the effective top-k and can return too few or zero usable chunks even when relevant, authorized content exists further down the ranked list.

This is one of the most common architecture mistakes in enterprise RAG: retrieve the top ten chunks by similarity, then strip out the ones the user isn't allowed to see, and ship whatever remains. If a user is authorized for only a fraction of the corpus, most of that top ten can legitimately belong to documents they can't see, leaving two or three chunks, or zero, to actually answer with — and the failure is invisible in testing with an admin account that can see everything. The correct pattern pushes the permission check into the retrieval query itself as a metadata filter — tenant ID, security group, document ACL — evaluated alongside the vector similarity search, so the database returns the top k among authorized documents, not the top k overall with authorization applied afterward. This requires every chunk to carry the permission metadata needed to evaluate that filter at query time, which means ingestion has to capture and propagate ACLs from the source system, and that propagation has to stay in sync when permissions change after a document is already indexed — a permission downgrade that doesn't reach the index promptly is a real data leak, not a cosmetic bug. For multi-tenant systems, a hard per-tenant partition, separate collections or a mandatory tenant filter with no code path that can omit it, is the safer default over a single shared collection with an optional filter a bug could skip.

public interface ISecureChunkRetriever
{
    // The allowed-document/ACL filter is applied inside the query itself,
    // not on the results returned from it.
    Task<IReadOnlyList<RetrievedChunk>> SearchAsync(
        string query, IReadOnlyCollection<string> allowedDocumentIds, int topK, CancellationToken ct);
}

What interviewers look for: explicitly rejecting post-retrieval filtering and explaining why it breaks — it silently reduces effective top-k — which is the detail that distinguishes candidates who have actually built multi-tenant RAG from those who haven't.

Common mistakes: filtering by permission after retrieval instead of during it, and letting permission metadata go stale when a source document's access control changes after ingestion.

Q9 RAG is supposed to reduce hallucination. What still causes it, and what mitigations actually work?#

Short answer: Hallucination in a RAG system comes from two distinct failure points — retrieval returning nothing genuinely relevant, or generation ignoring the context it was given — and the mitigations differ for each: better retrieval plus an explicit "insufficient context" escape hatch for the first, and grounding instructions, citation requirements and a post-hoc groundedness check for the second; no single prompt tweak fixes both.

Retrieval failure produces the more forgivable but still dangerous version: the corpus genuinely doesn't contain the answer, retrieval returns the closest-available-but-irrelevant chunks, and a model instructed to answer using the context will often do exactly that with whatever it was given rather than admit the context doesn't cover the question — the fix is an explicit instruction and, ideally, a relevance threshold that triggers an "insufficient information" response instead of a forced answer. Generation failure is more insidious: the right chunk was retrieved, sits right there in the prompt, and the model still contradicts or embellishes beyond it, because nothing about a chat model's training specifically rewards strict adherence to provided context over its own parametric knowledge. Mitigations that measurably help include requiring the model to cite which chunk supports each claim, which discourages ungrounded assertions and gives an auditable trail; keeping the context free of irrelevant chunks that dilute attention; and running a post-hoc groundedness check, the same technique from the evaluation question, as a runtime guard on high-stakes answers, not only as an offline metric. None of this reduces hallucination to zero; the honest framing for an interview is reducing the rate and catching what slips through, not claiming the problem is solved.

What interviewers look for: the retrieval-failure-versus-generation-failure distinction applied specifically to hallucination, with concrete mitigations rather than a vague "RAG prevents hallucination" claim, which is itself a weak answer.

Follow-up questions:

• How would you design the "insufficient context" path so it doesn't trigger too often on genuinely answerable questions?
• Would you run a groundedness check synchronously on every request, or only on a sample? What drives that choice?

Q10 How do you control and predict the cost of a RAG system as it scales to a large corpus and high query volume?#

Short answer: Cost has three largely independent drivers that need separate budgets — one-time or periodic embedding cost proportional to corpus size and re-embedding frequency, storage and query cost proportional to vector count and dimensionality, and per-query generation cost dominated by how many tokens of retrieved context are forwarded — and the biggest lever most teams underuse is retrieving and forwarding less context per query, not switching to a cheaper model.

Embedding cost is usually the smallest and most predictable piece; it scales with corpus size and re-embedding frequency, and re-embedding should be a deliberate, budgeted event rather than something that happens accidentally on every ingestion re-run. Storage and query cost scale with vector count and dimensionality, which is why the dimensionality and chunk-size decisions from earlier questions are also cost decisions: smaller chunks mean more vectors for the same corpus, and a higher-dimensional embedding model costs more to store and search per vector. The dominant, most volatile cost is almost always generation: every retrieved chunk that goes into the prompt is billed as input tokens on every query, so a system retrieving and forwarding ten chunks when four would do is paying roughly two and a half times more per query, independent of which model it calls — this is why reranking down to a tight, high-precision context is as much a cost control as a quality improvement. Caching identical or near-identical queries, capping the maximum context forwarded per request, and routing simple factual queries to a smaller, cheaper model while reserving a stronger model for genuinely complex synthesis are the levers that move the bill the most, and all three are architecture decisions made well before anyone reaches for a different provider's pricing sheet.

What interviewers look for: separating the three cost drivers explicitly and naming context volume, not model choice, as the biggest generation-cost lever — the answer that shows real production cost ownership.

Common mistakes: treating "switch to a cheaper model" as the primary cost lever while ignoring how much retrieved context is forwarded on every call, and not budgeting re-embedding as a real, recurring cost when the embedding model or chunking strategy changes.