An enterprise AI assistant is the system design interview where "just call the LLM" meets every constraint a real company actually has: documents that update daily, permissions that vary per user and per document, a legal team that wants citations, a finance team that wants a cost ceiling, and a security team that assumes someone will eventually try to make the assistant say something it shouldn't. Interviewers use this scenario at the architect level precisely because it has no single hard part; ingestion, retrieval quality, access control, observability and cost all have to work together, and a design that's excellent at retrieval but leaks one tenant's documents into another's answers has failed the interview regardless of how good its embeddings are. The ten questions below walk through the whole platform: the ingestion pipeline, retrieval and reranking, permission-aware search, the LLM gateway, caching, evaluation, observability, cost controls and multi-tenancy.

Q1 Design the ingestion pipeline for an enterprise RAG assistant that indexes documents from several internal systems, a wiki, a ticketing system and SharePoint, with different update frequencies and different permission models. What are the moving parts?#

Short answer: Split ingestion into per-source connectors that normalize each system's documents into a common shape, text, metadata, and, critically, the source system's permission information, feeding a shared pipeline that parses, chunks and embeds that normalized content. Drive each connector from its own change signal, a webhook, a change feed or scheduled polling, rather than one fixed schedule for every source, since a ticketing system and a rarely edited policy wiki have very different freshness needs.

Each connector's job is narrow: pull documents changed since the last run, and resolve and attach that document's access-control information, which users, groups or roles can see it, at ingestion time, not at query time, because that's the only point where you reliably have a clean mapping between the source system's native permission model and your own. The shared pipeline downstream doesn't need to know anything about SharePoint or a ticketing system's API; it receives a normalized source document, a URI, text, a last-modified timestamp and a resolved list of security principals, and does the same parse-chunk-embed-upsert work regardless of origin, which keeps the system extensible as sources are added. Drive each connector from a background worker on its own schedule or event trigger, track a per-source watermark, a timestamp or a change-feed cursor, so a restart resumes from where it left off instead of re-scanning everything, and treat a permission-only change, someone removed from a group with the document itself unchanged, as a change that still requires re-indexing, since stale permission metadata is a security bug, not a staleness inconvenience.

What interviewers look for: permission resolution placed explicitly at ingestion time, not deferred to query time, and a connector architecture that isolates source-specific logic from the shared pipeline.

Common mistakes: designing ingestion only around content changes and forgetting that a permission-only change also has to trigger re-indexing; assuming one polling schedule fits every source.

Follow-up questions:

  • How would you handle a source system that can't tell you what changed, only its current full state?
  • What happens to already-indexed chunks when a source document is deleted?

Q2 Walk through chunking and metadata design for enterprise documents specifically, contracts, long policy PDFs, ticket threads. What goes wrong with a naive fixed-size chunker?#

Short answer: A naive fixed-character chunker ignores document structure, so it can split a contract clause or a policy requirement across two chunks that are individually meaningless and get embedded that way permanently. The fix is chunking by structure first, headings, sections, list boundaries, and falling back to a token budget only within an already-coherent section, plus attaching metadata that lets you filter and audit, not just retrieve.

Contracts and policies benefit heavily from heading-aware splitting because their meaning is structural; a clause only makes sense together with its section heading. Ticket threads are a different shape entirely, a growing, timestamped conversation, where chunking by message boundary rather than raw character size preserves who said what, and including the ticket's resolution status as metadata lets you filter out threads closed as invalid before they ever reach retrieval. Metadata beyond source URI and chunk index should include, for an enterprise assistant specifically, the resolved security principals from ingestion, which is what permission-aware retrieval filters on, a document type or category for scoped search such as "only search policies, not tickets," and a version or content hash so a stale chunk can be identified and cleanly replaced rather than silently duplicated when a source document is re-ingested.

What interviewers look for: structure-aware chunking justified specifically for enterprise document types rather than a generic, repeated RAG answer, and security metadata named as something attached at chunking time, connecting this answer to ingestion and to permission-aware retrieval.

Common mistakes: chunking every document type identically regardless of its structure; omitting security metadata from the chunk record and trying to bolt permission filtering on somewhere else later.

Follow-up questions:

  • How would you chunk a large table inside a policy document without destroying its meaning?
  • How do you detect and avoid re-indexing a document that hasn't actually changed?

Q3 Design permission-aware retrieval, "security trimming," so the assistant never surfaces a passage the asking user isn't allowed to see. How would you implement this with Azure AI Search specifically?#

Short answer: Attach each indexed chunk's resolved security principals, user or group identifiers, to a filterable, non-retrievable field, and apply a filter built from the asking user's own group memberships to every query as part of the search call itself, never as a step applied to results after they come back, because filtering after retrieval means the sensitive content already left the index and reached application code that has to be trusted not to leak it.

Azure AI Search's documented pattern for this is a Collection(Edm.String) field marked filterable and not retrievable, populated with security principal strings such as Microsoft Entra group object IDs, queried with a filter such as security_ids/any(g:search.in(g, 'group1, group2')). Critically, the field is a plain string match; there is no authentication or authorization happening inside the search engine itself, so the correctness of the whole scheme depends entirely on your application resolving the asking user's real, current group memberships correctly before building that filter, every single time. Azure AI Search separately offers built-in document-level access-control-list support for search solutions that can't use this filter pattern, worth naming as an alternative even without going deep on it. The same principle applies regardless of vector store: a Filter expression on a search call in Microsoft.Extensions.VectorData plays the identical role for other backends, and the filter must run as part of the search call, on indexed fields, not as an in-memory check on results already materialized in application code, both for correctness and because post-filtering after an approximate nearest-neighbor scan can leave too few results when many matches get filtered out.

C#
var options = new VectorSearchOptions<DocChunk>
{
    // Every one of the caller's current group IDs, resolved fresh per request, never cached
    // for longer than the group membership itself might realistically change.
    Filter = c => c.TenantId == tenantId && c.SecurityIds.Any(id => callerGroupIds.Contains(id)),
};

await foreach (var hit in chunks.SearchAsync(question, top: 20, options, cancellationToken))
{
    candidates.Add(hit.Record);
}

What interviewers look for: filtering placed inside the search call on indexed fields, explicitly not as a post-retrieval step, and the specific, correct caveat that the security field is a plain string match with no real authorization behind it; the application is fully responsible for supplying the right principal list.

Common mistakes: retrieving broadly and filtering in application code afterward, which briefly brings unauthorized content into a process that then has to be trusted not to leak it; caching a user's group memberships so long that a removed permission stays effectively granted.

Follow-up questions:

  • How would you handle a user whose group memberships changed mid-session?
  • What would you log to prove, after the fact, that a specific answer only used documents the user could see?

Q4 Design the retrieval and reranking stage for an enterprise assistant answering from a large, heterogeneous corpus. Why isn't top-k vector search by itself good enough?#

Short answer: Retrieve a wide candidate set with hybrid, vector plus keyword, search so exact tokens such as contract numbers or ticket IDs aren't lost to embedding similarity alone, apply the security and tenant filters as part of that same search call, and then rerank the surviving candidates with a model or a semantic ranker that looks at the query and each candidate together, since vector similarity alone frequently ranks a topically related but unhelpful passage above the one that actually answers the question.

A practical shape: retrieve the top 20 to 30 candidates by hybrid search after filtering, then rerank down to the 3 to 8 chunks that actually go in the prompt. Azure AI Search's semantic ranker is a managed option for that second pass; a cross-encoder model or a fast chat model used as a judge, scoring each candidate against the query and keeping only the top few, is a workable do-it-yourself alternative that needs no extra infrastructure. Capping the pre-rerank candidate count matters more here than in a single-tenant system, because the security filter itself can shrink the eligible set unpredictably per user; retrieval logic needs to handle "the top 20 after filtering happens to be thin for this particular user" gracefully rather than assuming a fixed-size candidate set always arrives.

What interviewers look for: the retrieve-wide-then-rerank-narrow pattern applied specifically to a filtered, permission-scoped candidate set, not a generic, unfiltered RAG answer, and awareness that filtering can shrink the candidate pool unevenly across users.

Common mistakes: applying reranking before the security filter, wasting work reranking content the user can't see and risking a leak through the reranker's own output; assuming the candidate pool is always full-sized regardless of who's asking.

Follow-up questions:

  • What would you do if a specific user's filtered candidate set is consistently too thin to answer well?
  • Would you rerank differently for a compliance-sensitive query than for a casual one?

Q5 Design the LLM gateway layer that sits between your application services and the underlying model providers. What does it centralize, and why not just call the model SDK directly from every service?#

Short answer: A gateway centralizes the concerns that are identical across every feature that calls a model: authentication to the provider, per-tenant rate limiting and budget enforcement, request and response logging with redaction, routing to the right model or deployment, and response and prompt caching. Every individual service then calls one internal, stable interface instead of every team independently reimplementing throttling, retries and cost tracking, usually inconsistently.

In a .NET architecture, this is naturally built as IChatClient middleware through the ChatClientBuilder pipeline when the logic can live in-process, or as an actual proxy service, such as Azure API Management's dedicated AI gateway policies, when the concern genuinely needs to be enforced centrally across many independently deployed services that don't share a process. The middleware pipeline pattern composes cleanly: function invocation for the tool loop, a custom layer for input screening, distributed caching for exact-match responses, OpenTelemetry for the GenAI semantic-convention traces, and a resilience layer for retries on throttling, each concern isolated to one component instead of scattered through application code, and swappable, a different judge model, a different cache backend, without touching callers. Calling the provider SDK directly from every service is exactly how a platform ends up with inconsistent retry logic, no unified view of spend, and a prompt-injection defense that only some features remembered to add.

What interviewers look for: the gateway justified by what it centralizes and keeps consistent across teams, and a concrete .NET shape for it, middleware pipeline versus a genuine network proxy, rather than treating "gateway" as an unexplained black box.

Common mistakes: describing a gateway only as "a place that calls the model provider" without naming what it actually centralizes; picking a network-hop proxy for everything even where in-process middleware would add the same behavior with less latency.

Follow-up questions:

  • When would you choose a real network-hop gateway over in-process middleware?
  • How would the gateway handle two teams needing different content-safety policies for the same underlying model?

Q6 What should and shouldn't be cached in this platform, and why does treating an LLM response like any other cacheable API call go wrong?#

Short answer: Cache the parts that are genuinely repeated with identical or near-identical input, retrieved chunks and embeddings for popular queries, exact-match full responses for templated or classification-style calls, and don't rely on caching for open-ended conversational answers, where two users almost never send identical input, so an exact-match cache's hit rate there is close to zero regardless of how it's implemented.

Three distinct caching layers apply, and they solve different problems. Exact-match response caching skips the model entirely on an identical repeat call, but delivers close to zero value for open chat. Provider-side prompt caching reuses previously processed prefix tokens, a long, unchanging system prompt or a large retrieved document, across requests that share that prefix; you don't implement it directly, you structure prompts with the stable, shared part first and the variable, per-request part last, and observe the effect through token-usage telemetry. Semantic caching, matching near-duplicate questions by embedding similarity, is genuinely useful for a high-traffic, FAQ-style assistant, but it's a higher-risk technique: a similarity threshold that is too loose returns a wrong cached answer confidently and silently, so it belongs only on low-stakes, easily verified answer types, with cache hits logged distinctly from fresh model calls so they can be audited. For the RAG-specific pieces, caching embeddings for repeated or popular queries is close to free value, since embedding the same question twice is pure waste, while caching retrieval results has to respect the same per-user security filter as a live query; caching an unfiltered retrieval result and reusing it across users would quietly reintroduce the exact leak permission-aware retrieval was designed to prevent.

What interviewers look for: the three distinct caching layers correctly distinguished with the right use case for each, and the specific trap of caching retrieval results across users without respecting per-user security filtering.

Common mistakes: expecting meaningful hit rates from exact-match caching on open-ended chat; caching a retrieval result keyed only by the question text, ignoring that the same question from two different users can have two different permitted answer sets.

Follow-up questions:

  • How would you cache embeddings safely without ever caching the retrieved content itself?
  • What would make you disable semantic caching for a specific feature entirely?

Q7 How do you evaluate whether this platform is actually working, both before shipping a change and continuously in production?#

Short answer: Evaluate retrieval and generation as two separate, measurable properties, did the retrieval step find the right passages, and did the answer actually stay faithful to them, using a curated, versioned set of representative questions with known-good answers, run on every meaningful pipeline change and again continuously against sampled production traffic, because an LLM-based system's quality can silently drift even when no application code changed, purely from a model or provider-side update.

The Microsoft.Extensions.AI.Evaluation.Quality package ships evaluators for exactly this split: a retrieval evaluator scores how well the retrieved chunks serve the query, a groundedness evaluator scores whether the answer stays faithful to the retrieved context, and a relevance evaluator scores whether it actually addresses the question. Each uses an LLM as judge and returns a score with a rationale, which is what makes it possible to spot, for instance, a low relevance score paired with a high retrieval score and correctly conclude the bug lives in the prompt or the generation step, not the index. For an enterprise assistant specifically, the evaluation set needs deliberately adversarial and edge-case entries alongside the happy path, questions with no good answer in the corpus, to verify the system says so instead of guessing, and, given the security stakes here, a specific evaluation lane that verifies a user without access to a document never receives an answer grounded in it, a functional test of the permission-aware retrieval design, not just an answer-quality metric.

What interviewers look for: retrieval and generation evaluated as separate, distinctly measurable properties, and a security-specific evaluation lane named explicitly rather than assuming answer-quality evaluation alone would ever catch a permission leak.

Common mistakes: evaluating only end-to-end answer quality with no way to tell whether a bad answer came from bad retrieval or bad generation; never testing that permission filtering actually holds under automated evaluation, only under manual review.

Follow-up questions:

  • How would you build an adversarial test specifically for the security-trimming boundary?
  • How often would you re-run the full evaluation suite, and what would trigger an out-of-cycle run?

Q8 Design observability for this platform. What do you need to see to debug a bad answer a user complained about three days ago?#

Short answer: Trace and store enough of the request's shape, which documents were retrieved and which were filtered out, which reranker scores they received, the assembled prompt or a redacted version of it, the model and its response, and the token usage and cost, tagged with the request's tenant, feature and a correlation ID. Use the OpenTelemetry GenAI semantic conventions as the common shape so this information is queryable, while keeping raw prompt and completion content off by default and behind a deliberate, access-controlled opt-in.

Instrument every model call with OpenTelemetry, which emits standardized attributes for the operation, the model, and input and output token counts, and tag each call with your own feature and tenant dimensions, since the model provider has no concept of either. Extend that same discipline specifically to the RAG pipeline's own steps: instrument retrieval as its own span, which chunks came back, their scores, how many survived the security filter, and reranking as another, so a three-day-old complaint can be reconstructed from stored trace data, which documents were even eligible for this user, which ones the search actually returned, which ones survived reranking into the prompt, without needing to reproduce the bug live. For a permission-related or retrieval-quality bug specifically, that stored trace is often the only way to distinguish "the right document was never retrieved" from "it was retrieved but the model ignored it" after the fact.

What interviewers look for: the retrieval and reranking steps named as their own instrumented spans, not just the final model call, since debugging a RAG-specific failure needs visibility into what did and didn't make it into the prompt, and content capture correctly scoped as opt-in.

Common mistakes: only instrumenting the final chat completion call, with no record of what retrieval actually returned, which makes a three-day-old complaint nearly undebuggable; turning on full content capture in production without a redaction and retention plan.

Follow-up questions:

  • How long would you retain retrieval traces versus raw prompt content, and why might those retention periods differ?
  • How would you correlate a user's complaint back to a specific trace without asking them for a timestamp?

Q9 Design cost controls for a platform used by many internal teams, where one team's careless prompt, or a bug in a tool-calling loop, could otherwise blow the monthly budget.#

Short answer: Combine a fast, synchronous gate that checks accumulated spend against a per-tenant budget before an expensive call is allowed to proceed, with a slower feedback loop of dashboards and alerts on cost tagged by tenant and feature that catches gradual drift a hard per-call gate would miss, and route routine, low-difficulty work, classification, extraction, to a smaller, cheaper model by default rather than defaulting every call to the platform's largest model.

The budget gate's degrade-gracefully behavior matters a lot in a multi-tenant setting: when a tenant hits its daily limit, the platform should return a smaller-model answer, a cached answer, or a clear "budget exceeded, try again tomorrow" error, not a raw failure, so a runaway cost problem in one team doesn't turn into an outage-shaped incident for that team's users. Explicitly call out the tool-calling or agent loop as the most common real-world cause of a cost spike on a platform like this; a bounded limit on the number of tool-invocation iterations per request is a cheap, high-value safety valve that budgets alone won't substitute for, since a runaway loop can burn through a daily budget in minutes, well before a human notices a dashboard trend. Static routing by endpoint, deciding at development time which features need the large model and which can use the small one, is simpler and cheaper to reason about than a dynamic router that classifies task difficulty at runtime, and is enough for most platforms.

What interviewers look for: a fast, per-call gate distinguished from a slower dashboard and alerting loop, each solving a different failure speed, and the tool-loop runaway named specifically as the most common real cause of a cost spike on an agentic platform.

Common mistakes: relying only on end-of-month billing dashboards to catch a cost problem, by which point the responsible team and feature are hard to reconstruct; having no bound on tool-calling iterations, leaving budgets as the only, too-slow, backstop against a runaway loop.

Follow-up questions:

  • How would you attribute cost fairly when a request touches a shared component used by multiple tenants' calls?
  • What would you do differently for a tenant that legitimately needs a higher budget than the platform default?

Q10 How do you design multi-tenancy into this platform from day one, so a new enterprise customer's data can never leak into another's answers or evaluation data?#

Short answer: Treat tenant isolation as a property enforced independently at every layer, a tenant identifier on every ingested chunk used as an indexed filter field, a tenant-scoped or wholly separate vector store where isolation requirements demand it, tenant tags on every cost and observability metric, and a tenant-scoped evaluation set and cache, rather than assuming that getting it right in one layer, say the retrieval filter, automatically protects every other layer that also touches tenant data.

This question ties the whole design together. Multi-tenancy is not a single feature to add; it has to be independently true in ingestion, which tenant a document belongs to, resolved and stored at write time, the same as security principals; in retrieval, the same indexed tenant-filter pattern as permission-aware retrieval, since tenant isolation and permission-aware retrieval are the same mechanism applied at two different granularities; in caching, where reusing a retrieval result across users applies identically across tenants, and more severely, since a cross-tenant cache leak is a much bigger incident than a cross-user one inside the same company; in observability, where per-request tagging needs a tenant dimension so a dashboard, an alert or a support investigation can be scoped to one customer; and in evaluation, where the evaluation set and any stored transcripts used for it need to respect the same tenant boundaries as production data, since a shared evaluation dataset that mixes tenant content is itself a leak. For customers whose contract or regulatory requirements demand it, a fully separate vector store, or even a fully separate deployment, per tenant is the strongest guarantee available, at real operational cost; a shared store with a rigorously enforced, indexed tenant filter is what most platforms run for everyone else, and the architectural discipline required to keep that filter correct everywhere, always, is the actual hard part.

What interviewers look for: tenant isolation connected explicitly back to the earlier answers, ingestion, retrieval, caching, observability, evaluation, as one consistent property enforced independently at each layer, rather than treated as a single filter that, once added anywhere, is assumed to protect everything.

Common mistakes: enforcing tenant isolation in retrieval but overlooking it in caching or in a shared evaluation dataset; assuming a single filter fix covers every layer the data passes through.

Follow-up questions:

  • What would push you toward fully separate infrastructure per tenant instead of a shared, filtered store?
  • How would you test, specifically, that tenant isolation holds, not just that it's implemented?

Quick-Fire Round#

QuestionAnswer
Where should document permissions be resolved?At ingestion time, attached to each chunk, not computed at query time.
What field type does Azure AI Search use for security trimming?A filterable, non-retrievable Collection(Edm.String) field of security principal IDs.
Does Azure AI Search's security field enforce authorization itself?No; it's a plain string match, and the application must supply the correct principal list.
Should security filters run before or after reranking?Before; reranking unfiltered results wastes work and risks leaking signal about hidden content.
Why is exact-match response caching nearly useless for open chat?Two users almost never send identical input, so the hit rate is close to zero.
What's the most common cause of a runaway cost spike on an agentic platform?An unbounded tool-calling loop, not a single expensive prompt.
What are the two properties a RAG evaluation suite must measure separately?Retrieval quality and answer groundedness.
What's the strongest tenant-isolation guarantee available?A fully separate vector store or deployment per tenant.
Should retrieval results be cached across different users?Not without re-applying the same per-user security filter, or it reintroduces the leak filtering prevents.

How to Prepare#

  • Be able to state precisely where in the pipeline permission resolution happens, ingestion, and where the filter is applied, inside the search call, before reranking; this single thread runs through most of the hard questions in this interview.
  • Know the concrete mechanism for security trimming in at least one real vector store; Azure AI Search's filterable security field is a strong, verifiable example, rather than a hand-wavy "we'd filter by permissions."
  • Practice separating retrieval evaluation from generation evaluation, and be ready to name a security-specific evaluation lane, not just answer-quality metrics.
  • Have a clear, layered answer for cost control: a fast per-call budget gate, a slower dashboard and alerting loop, and a bounded tool-calling loop as three distinct defenses.
  • Rehearse tying multi-tenancy back through every earlier answer, ingestion, retrieval, caching, observability, evaluation, as one consistent property, not a single filter.