CAP, Consistency Models and Idempotency Interview Questions

Q1 State the CAP theorem precisely. What do "consistency," "availability" and "partition tolerance" actually mean, and why is the common "pick two of three" framing misleading?#

Short answer: CAP says that when a network partition occurs, a distributed system storing replicated data must choose between consistency — every read sees the most recent write, or an error — and availability — every reachable node returns a non-error response. Partition tolerance isn't really a choice, because a system that can't tolerate partitions isn't a distributed system in any meaningful sense; the real decision CAP describes is CP versus AP during a partition, not a permanent, always-in-effect trade-off across all three.

Consistency in CAP means linearizability: a read that starts after a write completes must see that write, as if there were only a single copy of the data, no matter which replica served it. Availability means every node that's still reachable must return a valid, non-error response to every request, even if it can't confirm that response reflects the latest write. Partition tolerance means the system keeps functioning, in some form, when network messages between nodes are dropped or delayed. The "pick two of three" framing is misleading because partition tolerance isn't optional for any system running on more than one machine — networks partition whether you design for it or not, so the real question CAP forces is what a node does when it can't reach its peers: serve a possibly stale, or outright refuse to serve, response and stay available, or refuse to answer until it can guarantee correctness and lose availability.

It's also misleading because CAP is not a statement about normal operation — most of the time there's no partition, and a well-designed system can be both consistent and available simultaneously. CAP only bites during the partition itself, which is exactly the gap PACELC closes: it separates what a system does during a partition from what it trades off the rest of the time, which is the question CAP alone doesn't answer and the one architects actually face most often.

What interviewers look for: precision — linearizability, not "everyone sees the same data eventually" — and the correction that partition tolerance isn't a real choice, which separates candidates who've internalized the theorem from those repeating a slogan.

Common mistakes: describing CAP as an always-in-effect, permanent trade-off rather than a statement about behavior specifically during a partition.

Q2 What is PACELC, and why is it a more useful model than CAP alone for a system that's running normally, with no partition?#

Short answer: PACELC extends CAP: if there is a Partition, choose between Availability and Consistency, which is the CAP part; Else, during normal operation with no partition, choose between Latency and Consistency. It's more useful because most of a distributed system's life is spent in the "else" branch, and CAP has nothing to say about the trade-off that actually dominates day-to-day design: how much latency you accept to get a stronger consistency guarantee on every request.

The "else" branch is where most architectural decisions actually live. A system that replicates synchronously to a quorum before acknowledging a write gets strong consistency, but every write pays the latency cost of a round trip to multiple replicas, potentially across availability zones or regions. A system that acknowledges a write as soon as it hits the primary and replicates asynchronously gets lower latency, but a read against a replica can return stale data with no partition anywhere in sight — the replica simply hasn't caught up yet. This is a decision made on every deployment regardless of whether a partition has ever happened, which is why PACELC is the model senior engineers reach for when asked to justify a specific database or replication configuration.

In practice this shows up constantly in .NET architectures: a globally distributed database account configured for strong consistency accepts higher write latency and reduced availability during a regional outage in exchange for linearizable reads everywhere; the same account configured for a weaker consistency level trades that away for lower latency and higher availability. Neither choice is more correct — PACELC is the vocabulary for explaining precisely which one you picked and why, instead of vaguely gesturing at "consistency versus performance."

What interviewers look for: recognizing that PACELC's else-branch trade-off, not the partition branch, is what actually governs most day-to-day design decisions, and connecting it to a concrete configuration choice in a real datastore.

Q3 What is eventual consistency, and how do you design a user-facing feature so users don't perceive it as a bug?#

Short answer: Eventual consistency guarantees that if no new writes occur, all replicas will eventually converge to the same value, with no bound on how long "eventually" takes unless the system specifically provides one. The UX problem it creates is a user performing an action and then immediately seeing state that doesn't reflect it, which reads as broken software even when the system is behaving exactly as designed.

The most common failure is a user submitting a change — updating a profile, posting a comment, placing an order — and then being shown a view that reads from a replica or cache that hasn't caught up, so their own action appears to have silently failed. The fix is almost never "make the whole system strongly consistent," since that trades away the latency and availability benefits eventual consistency exists to provide, for a problem that's really about one specific read path, not the whole system. Instead, the standard pattern is optimistic UI: apply the change to client-side state immediately, on the assumption it will succeed, and reconcile silently if the server later disagrees, so the user never sees a window where their own action looks like it didn't happen.

Where a full round trip to the server is unavoidable, route the specific read that follows a write back to the primary, or to a replica guaranteed to have seen it — this is read-your-writes consistency, covered next — rather than the system's default, load-balanced read path. For data other users will see, such as a comment appearing in someone else's feed, eventual consistency is usually fine to expose honestly, sometimes with a visible cue like a relative timestamp or a pending state, that sets the right expectation instead of pretending the system is instantaneous when it structurally isn't.

What interviewers look for: the insight that eventual-consistency UX problems are almost always about the read path immediately following a user's own write, not the system's consistency model as a whole, and a specific pattern rather than "just show a loading spinner."

Follow-up questions:

• How would you handle a write the optimistic UI predicted correctly, versus one the server later rejects?
• What's the UX difference between eventual consistency on a user's own data versus on data other users produced?

Q4 Explain read-your-writes consistency and how you'd implement it against a primary/replica database or an eventually consistent cache.#

Short answer: Read-your-writes guarantees that once a specific client has performed a write, that same client's subsequent reads will always reflect it, even if other clients might still see stale data from a replica that hasn't caught up. It's a per-client guarantee, not a system-wide one, which is exactly what makes it cheap enough to provide without abandoning eventual consistency everywhere else.

The simplest implementation is sticky routing: after a write, route that client's subsequent reads to the primary, or to the specific replica confirmed to have applied the write, for some window of time, instead of the load balancer's normal replica-selection logic. A common variant tracks a logical write position — a sequence number, a replication log offset, or a timestamp — returned to the client, often embedded in a session token, after a write; on the next read, the client presents that position, and the system either routes to a replica known to have reached it or makes the serving replica wait until its own applied-log position catches up before answering.

public sealed record WriteResult(long CommitPosition);

public async Task<Order> GetOrderAsync(string orderId, long? readAfter, CancellationToken ct)
{
    var replica = readAfter is null
        ? _replicaSelector.PickAny()
        : await _replicaSelector.PickCaughtUpToAsync(readAfter.Value, ct);

    return await replica.GetOrderAsync(orderId, ct);
}

The cache case is structurally the same problem: after a write-through update, either invalidate the cache entry synchronously as part of the write, so the next read is a guaranteed miss that goes to the source of truth, or, for an asynchronously updated cache, tag the cached value with the write position and have the read path reject a stale entry the same way it would reject a lagging replica. The key design decision either way is scope: read-your-writes only needs to hold for the client that performed the write, so the routing or freshness-check cost lands on a small fraction of traffic, not on every read in the system.

What interviewers look for: the distinction between a per-client and a system-wide guarantee, and a concrete mechanism, such as sticky routing or a tracked write position, rather than a hand-wavy "check the cache is fresh."

Q5 Why do idempotency keys matter for APIs that can be retried, and how would you design an idempotency-key mechanism for a payment or order-creation endpoint?#

Short answer: Any operation that isn't naturally idempotent, like creating an order or charging a card, becomes dangerous the moment a client can retry it, because a retry might be caused by a timeout on a request that actually succeeded server-side. An idempotency key lets the client label a logical operation once, so the server can recognize a retry of the same operation and return the original result instead of executing it again.

The client generates a unique key, typically a GUID, once per logical operation, before the first attempt, and sends it on every retry of that same operation. The server, on receiving a request with an idempotency key, checks a durable store — the same transaction as the business write, not a separate cache — for that key: if it's new, it executes the operation and atomically records the key alongside a description of the response; if the key already exists, it returns the stored response without re-executing anything. The atomicity between doing the work and recording the key is the entire mechanism — if those two things can happen non-atomically, you've reintroduced the exact race the key was meant to close.

public async Task<IResult> CreateOrderAsync(
    CreateOrderRequest request, string idempotencyKey, CancellationToken ct)
{
    var existing = await _store.FindByKeyAsync(idempotencyKey, ct);
    if (existing is not null)
    {
        return Results.Json(existing.ResponseBody, statusCode: existing.StatusCode);
    }

    await using var tx = await _db.BeginTransactionAsync(ct);
    var order = await _orders.CreateAsync(request, ct);
    await _store.RecordAsync(idempotencyKey, order, ct); // same transaction
    await tx.CommitAsync(ct);

    return Results.Created($"/orders/{order.Id}", order);
}

A key detail interviewers probe: an idempotency key scopes one specific logical request body, not just an endpoint — reusing a key with a different payload should be rejected, not silently return the first response for a different-looking request. Keys also need a retention window, long enough to cover realistic retry and backoff periods but short enough not to grow the table forever, and, for endpoints where two genuinely different requests could race with the same key concurrently, a short-lived lock or a unique constraint on the key column so a concurrent duplicate insert fails loudly instead of executing twice.

What interviewers look for: the atomicity requirement specifically, and awareness of the request-body-mismatch and concurrent-duplicate edge cases, not just "generate a GUID and check if you've seen it."

Common mistakes: storing the idempotency key in a separate cache from the business data, which reopens a race window between checking the cache, doing the work and updating the cache that the whole mechanism exists to close.

Q6 What's the difference between at-most-once, at-least-once and exactly-once delivery, and why is "exactly-once" usually really "effectively-once"?#

Short answer: At-most-once means a message is delivered zero or one times, never redelivered, so failures cause silent loss; at-least-once means a message is delivered one or more times, never lost but possibly duplicated; exactly-once means delivered precisely once, with no loss and no duplication — provably impossible to guarantee end-to-end across an unreliable network without cooperation from the consumer, which is why systems that advertise it are actually providing at-least-once delivery plus deduplication, commonly called effectively-once.

The impossibility isn't an implementation gap that better engineering fixes — it follows from an unreliable network's inability to distinguish "the message was lost" from "the message arrived and only the acknowledgment was lost," so a producer must either retry, risking a duplicate, or not retry, risking silent loss. Every real messaging system picks a side of that trade-off at the transport level and then, if it wants exactly-once-like semantics, closes the gap with idempotency at the consumer: the broker guarantees at-least-once delivery, and the consumer deduplicates, using a message ID, an idempotency key, or a natural business key, so processing the same message twice produces the same side effect as processing it once.

This is the same idempotency mechanism as the payment endpoint in the previous question, just triggered by message redelivery instead of an HTTP retry: track processed message IDs in the same transaction as the side effect they cause, so a redelivered message either does nothing, if already processed, or does the work and records the ID atomically. Transactional producer and consumer features in modern brokers implement this pattern internally, scoped to that broker's own topics — they don't extend the guarantee to an external side effect like a database write or an HTTP call your consumer makes, which is why you still need to design that boundary yourself in any real system.

What interviewers look for: stating plainly that true exactly-once across an arbitrary network boundary is impossible, and connecting effectively-once to the same idempotent-consumer pattern used for retried API calls, rather than treating it as an unrelated messaging-specific trick.

Q7 Why can't distributed systems rely on wall-clock time to order events, and how do logical clocks solve the problem?#

Short answer: Wall clocks on different machines drift and are only ever synchronized to within some margin of error — even clocks disciplined by time-synchronization protocols can differ by measurable amounts — so two events on different nodes that happen close together in real time can have timestamps in the wrong order, or identical timestamps. Logical clocks like Lamport timestamps sidestep the problem by tracking causality — did event A happen-before event B — instead of trying to measure real elapsed time at all.

A Lamport timestamp is a counter each node maintains: on any local event, increment the counter; on sending a message, attach the current counter value; on receiving a message, set the local counter to the maximum of the local and received values, plus one. This produces a partial order that guarantees if A causally happened-before B — A is the cause, or on the causal path that led to B — then A's timestamp is strictly less than B's. It does not guarantee the reverse, so two genuinely concurrent, causally unrelated events can end up with either ordering, or even the same timestamp, which is fine, because there was never a real ordering requirement between them to begin with.

public sealed class LamportClock
{
    private long _counter;

    public long Tick() => Interlocked.Increment(ref _counter);

    public long Receive(long incoming)
    {
        long observed, updated;
        do
        {
            observed = Volatile.Read(ref _counter);
            updated = Math.Max(observed, incoming) + 1;
        } while (Interlocked.CompareExchange(ref _counter, updated, observed) != observed);
        return updated;
    }
}

Where Lamport timestamps fall short is telling whether two events are causally related at all — they give a single number consistent with causality but collapse concurrent, unrelated events into an arbitrary total order. Vector clocks fix that: each node keeps a full vector of counters, one per node, and comparing two vectors tells you definitively whether one happened-before the other or whether they're genuinely concurrent, which is exactly the information conflict-resolution logic needs, at the cost of a vector that grows with the number of nodes instead of a single integer.

What interviewers look for: the specific happens-before relationship, not "logical clocks avoid clock drift" as a vague slogan, and clarity on what a Lamport timestamp cannot tell you, which is the reason vector clocks exist.

Follow-up questions:

• Why does a vector clock's size grow with the number of nodes, and what problem does that create at scale?
• How does a hybrid logical clock combine a physical timestamp with a logical counter, and why would you want that?

Q8 Walk through common conflict-resolution strategies when two replicas diverge: last-write-wins, application-level merge and CRDTs. What are the trade-offs?#

Short answer: Last-write-wins picks a winner using a timestamp or version number and silently discards the other write — simple, but it loses data whenever the discarded write mattered. Application-level merge calls domain-specific logic to combine both writes into a result that preserves intent from each side. CRDTs, conflict-free replicated data types, are data structures mathematically designed so any two divergent replicas merge deterministically and losslessly, without custom merge logic per field.

Last-write-wins is attractive because it requires no domain knowledge — attach a timestamp or a monotonically increasing version to every write, and on conflict, keep the higher one. Its failure mode is exactly what the name implies: if two users concurrently update different fields of the same record, or make genuinely compatible changes, last-write-wins still throws one of them away entirely, because it operates at the granularity of the whole write, not the semantics of what changed. It's a reasonable default where losing a rare, low-stakes conflicting write is acceptable, and a poor choice for anything where silent data loss is a real cost, such as inventory counts, financial balances or collaborative documents.

Application-level merge is what most systems actually need once last-write-wins' data loss becomes unacceptable: a shopping-cart merge that unions both sides' items instead of picking one cart, a counter tracked as a delta rather than an absolute value so both increments apply, a field-by-field three-way document merge. This requires writing and maintaining merge logic per data type, and that logic has to be genuinely commutative and associative, producing the same result regardless of the order replicas observe conflicting writes in, or you've built a subtler version of the same data-loss bug. CRDTs formalize that requirement: structures like grow-only counters, observed-remove sets and per-field last-write-wins registers are proven, by construction, to converge to the same state regardless of merge order, which is why systems that need automatic, correct-by-construction merging build on them instead of hand-rolling merge functions, at the cost of a much smaller set of operations allowed on the data and real complexity in composing the right structure for each field.

What interviewers look for: naming the actual failure mode of last-write-wins, silent data loss, rather than just "it's simple," and understanding CRDTs as a constrained, provably correct special case of application-level merge rather than a drop-in fix for arbitrary data.

Common mistakes: assuming last-write-wins is good enough without asking what specifically gets silently discarded on conflict, or assuming CRDTs replace an arbitrary data model instead of offering a small family of structures with specific, limited operations.

Q9 A network partition splits your cluster in two. Walk through what an AP system does versus what a CP system does, using a concrete example.#

Short answer: During the partition, an AP system keeps both sides serving reads and writes independently, accepting that they'll diverge and need reconciliation once the partition heals. A CP system has one side, typically whichever retains a quorum of nodes, continue operating normally, while the minority side refuses writes, and often reads, rather than risk serving or accepting data that can't be guaranteed consistent — sacrificing availability on that side until the partition heals.

Take an inventory-count service split across two data centers by a network partition. An AP design lets both sides keep accepting orders and decrementing their local view of stock, because refusing to sell during a partition is treated as worse than occasionally overselling; when the partition heals, a reconciliation step, often application-level merge, combines both sides' decrements, and the system corrects any oversell after the fact, typically by canceling or backordering the excess. A CP design instead routes inventory writes only through whichever side holds a quorum, commonly via a consensus protocol, so the minority side simply rejects order attempts during the partition rather than risk two sides independently selling the last unit of the same item; the trade-off is that legitimate customers on the minority side can't complete a purchase until connectivity is restored.

Leader election is the canonical CP example in the other direction: a system built on a consensus protocol deliberately refuses to have two leaders active during a partition, because split-brain leadership is exactly the correctness violation consensus exists to prevent — the minority partition simply can't elect a leader and stalls, which is availability sacrificed on purpose, not a bug. The decision isn't abstract: it's what it costs the business if a write is wrong versus what it costs if a write is refused, and different parts of the same system frequently land on different answers — inventory count tolerant of eventual reconciliation, payment authorization not tolerant of it at all.

What interviewers look for: a real example worked through in both directions, not just definitions, and the framing that the choice is a business-cost trade-off made per data type, not a single global architectural decision.

Q10 How do you design idempotency and consistency across a distributed workflow that spans multiple services — for example, using the outbox pattern together with idempotent consumers?#

Short answer: The transactional outbox pattern solves the "did my database write and my message publish happen atomically" half of the problem by writing the event to an outbox table in the same local transaction as the business change, then relaying it to the message broker asynchronously. Idempotent consumers solve the other half by making every downstream handler safe to run twice, since the outbox's at-least-once relay, and the broker's own at-least-once delivery, both mean duplicates are a certainty, not an edge case.

Without an outbox, a service that writes to its database and then publishes an event as two separate operations has an unavoidable gap: if it crashes or the broker is unreachable between the two, you either lose the event, because the database committed but the publish never happened, or, if you publish first, you can announce an event for a change that then fails to commit. The outbox pattern closes that gap by making the event write part of the same local database transaction as the business change — both commit together or neither does — and a separate relay process, polling the outbox table or reading the database's change log, delivers the row to the broker afterward, retrying until it succeeds and only then marking it relayed.

CREATE TABLE outbox_messages (
    id UNIQUEIDENTIFIER PRIMARY KEY,
    aggregate_id NVARCHAR(64) NOT NULL,
    event_type NVARCHAR(128) NOT NULL,
    payload NVARCHAR(MAX) NOT NULL,
    occurred_at DATETIME2 NOT NULL,
    relayed_at DATETIME2 NULL
);

That relay is guaranteed at-least-once by construction — a retry after a crash before relayed_at is set will republish a row the broker may have already received — which pushes deduplication downstream to every consumer, exactly like the earlier question on delivery semantics: each consumer tracks processed event IDs, often the outbox row's own ID, in the same transaction as whatever side effect it performs, so a redelivered event is a safe no-op. Chained across several services, this pattern is what lets you build a multi-step distributed workflow — order placed, inventory reserved, payment captured, shipment scheduled — where each hop is individually atomic and safely retryable, without a distributed transaction spanning all of them; the saga pattern is the higher-level choreography of exactly this chain, including how to compensate when a later step fails.

What interviewers look for: connecting the outbox pattern's guarantee, local atomicity plus at-least-once relay, to why idempotent consumption is mandatory downstream, not optional, and recognizing this as the same building block the saga pattern is composed from.