System Design Interview: Payment Processing System in .NET

Q1 What requirements and constraints shape a payment system's design before you write any code?#

Short answer: Correctness dominates every other concern — a payment system would rather be slow or briefly unavailable than silently wrong — so the guiding constraints are idempotency on every mutating operation, an immutable and auditable record of every money movement, and an explicit boundary around what touches raw cardholder data, since that boundary determines your PCI DSS scope for the entire system.

Start by separating the two workloads that get conflated in a naive design: the synchronous path where a customer is waiting (authorize a charge, show a result) and the asynchronous path where correctness matters more than speed (settlement, reconciliation, payout). The synchronous path needs to respond quickly and unambiguously, which pushes you toward doing the minimum necessary work inline — reserve funds with the payment service provider (PSP) and record intent — and pushing everything else (ledger posting, notifications, fraud scoring beyond a fast pre-check) onto durable background processing. Regulatory constraints come next: PCI DSS governs anything that touches primary account numbers, and most architectures are explicitly designed to keep that surface as small as possible by never letting your own servers see raw card data at all. Finally, state the non-negotiable invariant out loud: every state transition — authorized, captured, refunded, failed — must be idempotent and traceable to a specific request, because the network between you and the PSP, and between your services, will fail in ways that produce retries, and a payment system that isn't safe under retries is not a correct payment system.

What interviewers look for: whether "idempotency first" is stated as a design principle before any component diagram is drawn, since it's the assumption every later answer needs to be consistent with.

Common mistakes: starting with a microservice diagram before naming the correctness and compliance constraints, or treating fraud detection and regulatory scope as add-ons rather than inputs to the initial architecture.

Q2 Why is idempotency the single most important property of a payment API, and how would you implement idempotency keys in ASP.NET Core?#

Short answer: Every mutating request can be retried by a client, a proxy, or your own code after a timeout, and if a retry can trigger a second real-world charge, the system is unsafe regardless of how well everything else is built; the fix is a client-supplied idempotency key on every state-changing request, stored server-side with the original response, so a retried request with the same key returns the original result instead of executing again.

Implement it as a dedicated table keyed by the idempotency key with a unique constraint, storing a hash of the request body alongside the stored response. On each request, look the key up first: if it exists and the stored request hash matches the incoming body, return the stored response without re-executing anything; if it exists with a different body, that's a client bug (key reuse across different requests) and should fail loudly rather than silently execute one of the two; if it doesn't exist, proceed, and persist the key and response in the same transaction as the side effect it protects, so a crash between "charge succeeded" and "key recorded" can't happen.

app.MapPost("/api/payments", async (
    [FromHeader(Name = "Idempotency-Key")] string idempotencyKey,
    ChargeRequest request, IPaymentService payments, CancellationToken cancellationToken) =>
{
    var existing = await payments.FindByIdempotencyKeyAsync(idempotencyKey, cancellationToken);
    if (existing is not null)
    {
        return existing.MatchesRequestHash(request)
            ? Results.Ok(existing.Response)
            : Results.Conflict("Idempotency key reused with a different request body.");
    }

    var result = await payments.ChargeAsync(idempotencyKey, request, cancellationToken);
    return Results.Ok(result);
});

What interviewers look for: the detail that the key and the side effect must be persisted atomically, and handling the "same key, different body" case explicitly instead of only the happy path.

Common mistakes: generating the idempotency key server-side (which defeats the purpose — the client needs to reuse the same key across its own retries) or checking for the key without also validating the request matches.

Q3 What does "exactly-once" really mean in a distributed payment flow, and why can't you fully achieve it?#

Short answer: True exactly-once delivery across an unreliable network is provably unachievable — you can't know whether a response was lost after the receiver already acted on the request — so real systems build "effectively-once" behavior instead: at-least-once delivery (retry until you get a confirmed response) combined with idempotent handling on the receiving side, so a duplicate delivery is harmless even though duplicates do occur.

This reframing matters because candidates who chase literal exactly-once semantics end up designing distributed transactions or two-phase commits across services that don't actually need them, at a cost in availability and complexity that doesn't buy real correctness — a coordinator can still crash mid-commit. The practical pattern is: make every operation idempotent (the previous question), make every multi-step workflow resumable from wherever it left off, and use an outbox table to guarantee that "the charge was recorded" and "the event announcing the charge was published" happen together — the application writes both the business change and the outbound event to the same local database transaction, and a separate relay process publishes the outbox rows to your message broker at-least-once, so a crash between the two never happens because they're never two separate steps to begin with. The receiving side of that event still has to deduplicate, because at-least-once delivery from the broker means the same event can arrive twice; see CAP, Consistency Models and Idempotency Interview Questions for how this generalizes beyond payments specifically.

What interviewers look for: correctly naming the theoretical limitation (you can't distinguish "processed but the ack was lost" from "not processed" without more information) and pivoting immediately to the practical mitigation, rather than either overclaiming exactly-once is achievable or treating the limitation as a reason to give up on correctness.

Q4 Design a double-entry ledger schema for this system. Why double-entry instead of a single balance column?#

Short answer: A single mutable balance column can be corrupted by a bug, a race condition, or a partial failure with no way to reconstruct what happened; a double-entry ledger records every money movement as a pair of entries — a debit and a credit of equal magnitude across two accounts — so the ledger is append-only, self-verifying (debits must always sum to credits), and gives you a complete, replayable history instead of a single number you have to trust blindly.

Model accounts (customer wallet, merchant payable, PSP clearing, fee revenue) and entries separately: an entry references a transaction ID, an account, a direction, and an amount in the currency's minor unit (cents, not a floating-point decimal, to avoid rounding error accumulating across millions of entries). A transaction is only valid if its entries sum to zero across accounts, which becomes a database constraint you can assert in code and periodically re-verify as an integrity check — genuinely powerful because a bug that violates the invariant is caught by the shape of the data, not just by testing.

CREATE TABLE LedgerEntries (
    EntryId       BIGINT IDENTITY PRIMARY KEY,
    TransactionId UNIQUEIDENTIFIER NOT NULL,
    AccountId     BIGINT NOT NULL,
    Direction     CHAR(1) NOT NULL CHECK (Direction IN ('D', 'C')),
    AmountMinor   BIGINT NOT NULL CHECK (AmountMinor > 0),
    Currency      CHAR(3) NOT NULL,
    CreatedAtUtc  DATETIME2 NOT NULL DEFAULT SYSUTCDATETIME()
);

A customer-facing "balance" is then a derived value — SUM(credits) - SUM(debits) for that account, optionally materialized into a cache or a snapshot table for read performance, but always reconstructable from the entries if the snapshot is ever in doubt. Entries are never updated or deleted; a correction is a new, opposite entry, which preserves the audit trail this question's answer to auditing depends on.

What interviewers look for: the "balance is derived, not stored as truth" framing, and the append-only, correction-via-reversal discipline, since it's the detail that shows real accounting-systems exposure rather than a generic database schema.

Common mistakes: modeling a single Balance column with in-place updates, or forgetting that the debit/credit invariant needs to be enforceable, not just documented as a convention developers are trusted to follow.

Q5 How would you integrate with an external payment service provider and handle its webhooks securely?#

Short answer: Make the outbound charge call idempotent (pass your idempotency key through to the PSP if it supports one) and treat its synchronous response as provisional; treat the PSP's asynchronous webhook — not the synchronous response — as the authoritative confirmation of final state, and verify every webhook's signature before trusting its payload, since an unauthenticated webhook endpoint is a direct path for an attacker to fabricate a "payment succeeded" event.

The synchronous call to the PSP can time out or fail without telling you whether the charge actually went through on their side — that ambiguity is unavoidable, which is exactly why the webhook exists as the source of truth: even if your synchronous call appears to fail, a later webhook can confirm the charge succeeded, and your reconciliation logic needs to treat that as valid, not as a duplicate to reject. Verify webhook authenticity with an HMAC signature check using a shared secret, comparing in constant time to avoid a timing side-channel, and process the payload asynchronously (acknowledge receipt fast, then hand off to a queue) so a slow downstream step doesn't cause the PSP to time out and retry, compounding the load.

app.MapPost("/webhooks/psp", async (HttpRequest request, IPspWebhookSecret secret,
    IPaymentEventQueue queue, CancellationToken cancellationToken) =>
{
    using var reader = new StreamReader(request.Body);
    var payload = await reader.ReadToEndAsync(cancellationToken);

    var signature = request.Headers["X-Signature"].ToString();
    var expected = Convert.ToHexString(
        HMACSHA256.HashData(secret.CurrentKey, Encoding.UTF8.GetBytes(payload)));

    if (!CryptographicOperations.FixedTimeEquals(
        Convert.FromHexString(signature), Convert.FromHexString(expected)))
    {
        return Results.Unauthorized();
    }

    await queue.EnqueueAsync(payload, cancellationToken);
    return Results.Ok();
});

Wrap the outbound PSP call itself in a resilience pipeline — retry with backoff for transient failures, a circuit breaker so a degraded PSP doesn't take your whole checkout path down with it — which Microsoft.Extensions.Http.Resilience provides via AddStandardResilienceHandler on the HttpClient registration; see Resilience in .NET with Polly and Microsoft.Extensions.Http.Resilience.

What interviewers look for: naming the webhook, not the synchronous response, as the authoritative signal, plus unprompted mention of signature verification — skipping it is a common and serious real-world vulnerability.

Common mistakes: trusting the synchronous API response as final state, or verifying webhook signatures with a non-constant-time comparison, which reintroduces the exact class of vulnerability the check exists to prevent.

Q6 Walk through a saga that handles a partial failure — for example, the PSP charge succeeds but your own database write fails immediately after.#

Short answer: Use an orchestrated saga with explicit states and compensations: reserve, charge, confirm, with a defined compensating action for every step that can partially succeed, and back every step with the outbox pattern so "the charge happened" and "the event announcing it" are never separated by a window where a crash can lose one but not the other.

Model the flow as a state machine — Started → PaymentAuthorized → Confirmed, with a Compensating branch reachable from any state where a downstream step fails — rather than as an ad hoc sequence of try/catch blocks, because the state machine is what lets a recovery process resume a stuck saga after a crash instead of leaving it in limbo. For the specific scenario asked — PSP charge succeeds, then your write fails — the fix isn't cleverness in the write path, it's making the write retryable and idempotent: persist "charge succeeded, PSP reference ID X" as its own durable step before attempting anything that depends on it, so a retry of that persistence step after a crash simply succeeds where it left off rather than re-charging the PSP. If a later step fails in a way that can't be retried into success — inventory unavailable, fraud hold triggered after the fact — the compensation is a refund or void request back to the PSP, issued with its own idempotency key, and the saga transitions to a terminal failed state with the reason recorded, never silently disappearing.

public enum PaymentSagaState
{
    Started, PaymentAuthorized, Confirmed, Compensating, Failed
}

What interviewers look for: distinguishing steps that are safely retryable from steps that need an explicit compensation, and insisting the saga's state is durably persisted rather than held only in memory or a message queue's redelivery.

Follow-up questions:

• How would you detect and recover a saga that's been stuck mid-flight for an unusually long time?
• Would you choose orchestration or choreography for this saga, and why?

Q7 How do you reconcile your internal ledger against the PSP's records, and how often should you do it?#

Short answer: Run an automated, scheduled reconciliation job — typically daily against the PSP's settlement report, and ideally a lighter-weight near-real-time check against its transaction API — that compares every internal ledger entry tied to a PSP reference against the PSP's own record of that transaction, and raises an alert on any discrepancy rather than silently accepting either side as correct.

Reconciliation exists because the webhook-as-source-of-truth design from an earlier question is still not infallible: a webhook can be lost entirely (not just delayed) if your endpoint is down during the PSP's retry window, or a bug can post a ledger entry that doesn't correspond to reality. The job pulls the PSP's settlement or transaction report for the period, joins it against your ledger by PSP reference ID, and classifies every mismatch into a small number of known categories — present in the PSP report but missing from your ledger (a lost webhook, needs backfilling), present in your ledger but missing from the PSP report (investigate immediately, this is the dangerous direction), or present in both with a mismatched amount. Implement this as a scheduled background service or a batch job outside the request path entirely, since reconciliation is explicitly not latency-sensitive, and its output should be a small number of actionable discrepancies with automatic backfill for the safe category and a paging alert for the dangerous one.

What interviewers look for: understanding reconciliation as a defense in depth mechanism that assumes the webhook pipeline is imperfect, not a formality performed because "that's what payment systems do."

Common mistakes: treating reconciliation as optional because webhooks "should" be reliable, or building it to only report mismatches without automatically resolving the safe, well-understood category of them.

Q8 What is PCI DSS scope, and how do you architect the system to minimize it?#

Short answer: PCI DSS scope covers every system component that stores, processes or transmits primary account numbers (PANs) or is connected to one that does; the architectural strategy for minimizing that scope is to never let your own servers see raw card data at all — use the PSP's hosted fields, drop-in UI, or client-side tokenization so the card number goes directly from the customer's browser to the PSP, and your backend only ever handles an opaque token.

This single decision is the one with the largest leverage on compliance cost: a system that touches raw PANs directly falls under the full, expensive set of PCI DSS requirements (network segmentation, extensive logging and monitoring, regular penetration testing, strict access control audits), while a system architected so cardholder data never transits or lands on your own servers can typically qualify for a substantially reduced self-assessment path, because the compliance burden shifts largely onto the PSP, whose entire business is built around meeting it. Concretely, that means: the checkout page embeds the PSP's hosted iframe or client-side SDK, which posts card details straight to the PSP and returns a token to your page's JavaScript; your backend receives only that token, uses it for the charge API call, and never has a code path, log statement or database column capable of holding a raw card number. Everything downstream of that boundary — the ledger, the saga, reconciliation — deals exclusively in tokens, transaction IDs and amounts, none of which are in PCI scope on their own.

What interviewers look for: the specific mechanism (client-to-PSP tokenization keeping raw PANs off your infrastructure entirely) rather than a vague "we'd be PCI compliant," and recognizing that scope reduction is a decision made at the architecture stage, not a checklist applied afterward.

Common mistakes: accepting raw card numbers on your own backend "just to pass them through" to the PSP, which pulls your entire request path into full PCI DSS scope for no architectural benefit.

Q9 What auditing and traceability does a payment system need, and how would you implement it in .NET?#

Short answer: Every state transition — every charge, refund, saga step and webhook received — needs an immutable audit record capturing what happened, who or what triggered it, and a correlation ID that ties it back to the originating request, because when money is involved "we think this is what happened" is not an acceptable answer during a dispute or a regulatory inquiry.

Implement this as an append-only audit event stream, written in the same transaction as the business change it describes (the same outbox-adjacent discipline used for the ledger), rather than as best-effort logging that can be lost or reordered. Each event should be self-contained enough to answer "what happened, to what, by whom, when, and as part of which request" without needing to join against mutable state that might have changed since.

public sealed record AuditEvent(
    Guid EventId,
    string EntityType,
    string EntityId,
    string Action,
    string ActorId,
    string CorrelationId,
    DateTimeOffset OccurredAtUtc,
    string PayloadJson);

Propagate a correlation ID from the originating HTTP request through every downstream call — the PSP request, the saga steps, the webhook that eventually confirms it — using System.Diagnostics.Activity and OpenTelemetry so a single trace ID ties the customer-facing request to every asynchronous consequence of it, which turns "reconstruct what happened to this payment" from a manual log-grepping exercise into a single trace lookup. Retention requirements for this data are typically longer than ordinary application logs and are often driven by regulatory or contractual obligations specific to your payment processing agreement, so treat audit retention as a deliberate policy decision, not a default log-rotation setting.

What interviewers look for: the same-transaction-as-the-change discipline for audit writes, and fluency with correlation IDs and distributed tracing as the mechanism that makes the audit trail actually usable across an asynchronous, multi-service flow.

Q10 A customer says they were charged twice. Walk through how you'd investigate, and what design choices prevent this from happening in the first place.#

Short answer: Start from the ledger and the audit trail, not the customer's statement: look up every transaction tied to their account or order ID in the reporting window, check whether there are genuinely two distinct PSP charge references or one charge that was double-reported; the design choices that should have prevented a real double charge are the idempotency key on the charge request and idempotent webhook processing, so the investigation is also a check on whether those controls actually worked.

Pull every ledger entry and audit event correlated to the order ID and lay them out chronologically: if there are two distinct entries with two distinct PSP reference IDs, that's a genuine double charge, and the next question is why the idempotency key didn't prevent it — a common real cause is a client retrying with a newly generated key instead of reusing the original one after a timeout, which defeats the mechanism entirely, so check the client's retry behavior alongside the server-side logs. If there's only one PSP charge but it shows up twice in a statement or a downstream system, that's usually a webhook processed twice without dedup, or a reconciliation job double-posting — both point at the ledger's idempotent-write guarantee (a unique constraint on PSP reference ID per ledger transaction) either being absent or having a gap. The remediation for a confirmed genuine double charge is a refund issued through the same saga-and-idempotency-key discipline as any other operation, not a manual out-of-band fix, so the correction itself is auditable and can't introduce a third inconsistency.

What interviewers look for: an evidence-first investigation that starts at the ledger and audit trail rather than the customer's phone call, and connecting the root cause back to a specific missing or bypassed control (idempotency key reuse, webhook dedup) instead of a vague "we'd look into it."

Common mistakes: proposing a manual database fix for the refund instead of running it through the same auditable, idempotent path as every other money movement.