Local AI in .NET means running language models and other neural networks on the user's machine or your own servers instead of calling a cloud API. The tooling has matured quickly: ONNX Runtime GenAI, Ollama, Foundry Local and the Windows AI APIs all expose small language models (SLMs) that .NET code can call, most of them through the same IChatClient abstraction you use for hosted models. This guide explains when local inference makes sense, how each option works, how to use GPUs and NPUs, what quantization does, and what performance to expect.

Why Run AI Locally?#

Hosted frontier models are more capable, so local AI has to earn its place. It does in five situations:

  • Privacy and data residency. Prompts and documents never leave the device or your network. This matters for regulated data, source code and anything a customer would not want sent to a third party.
  • Cost at volume. There is no per-token bill. High-volume, low-complexity tasks such as classification, extraction and summarization of short texts can be cheaper on hardware you already own.
  • Latency. No network round trip, and the first token can arrive quickly for short prompts, which suits interactive features in desktop and edge apps.
  • Offline operation. Field tools, factory floors, aircraft and laptops on bad connections keep working.
  • Control. You pin an exact model version. It never changes behavior under you, and it is never deprecated on a vendor's schedule.

The costs are real too. Small models reason less reliably than large hosted ones, and they need careful prompting and evaluation. Hardware varies widely across users, models are gigabytes to download, and you own updates, safety filtering and monitoring. A common architecture is hybrid: route simple or sensitive requests to a local model and complex ones to a hosted model, behind the same IChatClient interface. The Microsoft.Extensions.AI guide covers that abstraction.

How Local Inference Works#

A language model is a large file of weights plus a tokenizer. Generation has two phases. Prefill processes the whole prompt in parallel and is compute-bound. Decode produces one token at a time and is mostly limited by memory bandwidth, because the runtime reads most of the weights for every generated token. A key-value (KV) cache stores intermediate attention state so earlier tokens are not recomputed, and it grows with the context length.

Two model formats dominate local inference:

  • ONNX, run by ONNX Runtime and its generative extension, ONNX Runtime GenAI. This stack powers Foundry Local, Windows ML and the AI Toolkit for Visual Studio Code, and it reaches CPUs, GPUs and NPUs through pluggable execution providers.
  • GGUF, the llama.cpp format used by Ollama. Ollama wraps it in a model registry, a background service and a REST API on port 11434.

Your .NET code sits on top of either stack. Ideally, it talks to IChatClient and IEmbeddingGenerator, so switching runtimes or moving to a hosted model does not ripple through the application.

Choosing a Small Language Model#

Popular open-weight families that run well locally include Microsoft's Phi models (language and vision variants), Qwen, Llama, Mistral, Gemma, DeepSeek distillations and OpenAI's gpt-oss. All of them appear in the supported-architecture list of ONNX Runtime GenAI and in the catalogs of Foundry Local and Ollama. For speech, Whisper models handle transcription locally.

Choose by task and constraints, not by leaderboard position:

  • Size vs. hardware. Models from roughly one to a few billion parameters run on typical laptops, and larger models need a capable GPU or a lot of unified memory. Use the memory arithmetic in the quantization section below.
  • Capabilities. Check tool calling, structured output, vision input and context length for the exact model version. Support varies widely, and small models often call tools less reliably.
  • License. Open-weight licenses differ in commercial-use and attribution terms, so review them before you ship.
  • Your own evaluation. Build a small test set from real prompts and compare two or three candidates. Differences between models on your task matter more than generic benchmarks.

Getting Started with Ollama and OllamaSharp#

Ollama is the quickest way to try local models on Windows, macOS or Linux. Install it, pull a model, and it serves the model on http://localhost:11434:

Bash
ollama pull gemma4
ollama run gemma4 "Say hello in one sentence."

From .NET, use the community OllamaSharp package (5.4 at the time of writing). Microsoft deprecated its own Microsoft.Extensions.AI.Ollama preview package and recommends OllamaSharp instead. OllamaApiClient implements both IChatClient and IEmbeddingGenerator<string, Embedding<float>>:

C#
using Microsoft.Extensions.AI;
using OllamaSharp;

var endpoint = new Uri("http://localhost:11434");

IChatClient chat = new OllamaApiClient(endpoint, "gemma4");

var history = new List<ChatMessage>
{
    new(ChatRole.System, "You are a concise assistant for .NET developers."),
    new(ChatRole.User, "When should I use ValueTask instead of Task?"),
};

await foreach (ChatResponseUpdate update in chat.GetStreamingResponseAsync(history))
{
    Console.Write(update.Text);
}

// The same client type also generates embeddings, for example for local RAG.
// Use a dedicated embedding model that you have pulled; chat models embed poorly.
string embeddingModel = Environment.GetEnvironmentVariable("OLLAMA_EMBEDDING_MODEL")
    ?? throw new InvalidOperationException("Set OLLAMA_EMBEDDING_MODEL.");
IEmbeddingGenerator<string, Embedding<float>> embedder =
    new OllamaApiClient(endpoint, embeddingModel);
ReadOnlyMemory<float> vector = await embedder.GenerateVectorAsync("local vector search");

Because the code depends only on the abstractions, you can register the Ollama client in development and a hosted model in production with a configuration switch. You can also combine local embeddings with the vector stores described in the embeddings and vector databases guide. For shared development environments, the .NET Aspire Community Toolkit has an Ollama hosting integration that runs the server as a container.

Foundry Local: Microsoft's On-Device AI Runtime#

Foundry Local is Microsoft's end-to-end runtime for shipping on-device AI inside your own application. Its key features:

  • A curated catalog of optimized models, including Phi, Qwen, DeepSeek, Mistral, GPT OSS and Whisper.
  • Automatic hardware selection. It detects the device's NPU, GPU or CPU and downloads the best model variant for it.
  • Lifecycle management. It handles downloading, caching, loading and unloading models.
  • An optional OpenAI-compatible local web server, including support for the Responses API format.

It runs inference on ONNX Runtime. The C# SDK, Microsoft.AI.Foundry.Local, reached 1.0 in April 2026, and version 2.0.1 shipped in August 2026. On Windows, it can download and register hardware execution providers through WinML.

The native SDK runs inference in-process:

C#
using Betalgo.Ranul.OpenAI.ObjectModels.RequestModels;
using Microsoft.AI.Foundry.Local;
using Microsoft.Extensions.Logging.Abstractions;

await FoundryLocalManager.CreateAsync(new Configuration { AppName = "contoso-notes" },
    NullLogger.Instance);

var catalog = await FoundryLocalManager.Instance.GetCatalogAsync();
var model = await catalog.GetModelAsync("phi-3.5-mini")
    ?? throw new InvalidOperationException("Model not in catalog.");

await model.DownloadAsync(p => Console.Write($"\rDownloading {p:F0}%")); // cached after first run
await model.LoadAsync();

var client = await model.GetChatClientAsync();
var reply = await client.CompleteChatAsync([ChatMessage.FromUser("Name three uses of NPUs.")]);
Console.WriteLine(reply.Choices![0].Message.Content);

The SDK uses OpenAI-compatible request types rather than Microsoft.Extensions.AI. To plug Foundry Local into an IChatClient pipeline (with function invocation, caching and OpenTelemetry), start its local web service and point the official OpenAI .NET client at it:

C#
using System.ClientModel;
using Microsoft.AI.Foundry.Local;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Logging.Abstractions;
using OpenAI;

// Alternative startup: enable the OpenAI-compatible web service on a loopback address.
var config = new Configuration
{
    AppName = "contoso-notes",
    Web = new Configuration.WebService { Urls = "http://127.0.0.1:52495" },
};
await FoundryLocalManager.CreateAsync(config, NullLogger.Instance);
// ...get, download and load the model as above...
await FoundryLocalManager.Instance.StartWebServiceAsync();

IChatClient chat = new OpenAIClient(new ApiKeyCredential("unused"),
        new OpenAIClientOptions { Endpoint = new Uri(config.Web.Urls + "/v1") })
    .GetChatClient(model.Id)
    .AsIChatClient();

ONNX Runtime and ONNX Runtime GenAI in C#

ONNX Runtime (Microsoft.ML.OnnxRuntime, 1.30 at the time of writing) runs any ONNX model, such as classifiers, embedding models, vision models and models exported from PyTorch. ONNX Runtime GenAI (Microsoft.ML.OnnxRuntimeGenAI, 0.16) adds what language models need on top: tokenization, the generation loop, KV cache management, sampling and constrained decoding. It also ships an IChatClient implementation:

C#
using Microsoft.Extensions.AI;
using Microsoft.ML.OnnxRuntimeGenAI;

// A folder with genai_config.json, the ONNX weights and tokenizer files.
using IChatClient chat = new OnnxRuntimeGenAIChatClient(@"models\phi-3.5-mini-int4-cpu");

var options = new ChatOptions { MaxOutputTokens = 300, Temperature = 0.2f };

await foreach (var update in chat.GetStreamingResponseAsync(
    "Explain the difference between prefill and decode in two sentences.", options))
{
    Console.Write(update.Text);
}

The client applies the model's chat template, maps MaxOutputTokens, Temperature, TopP, TopK, Seed and stop sequences to generator options, and streams tokens as they are produced. When you need full control, for example over a custom sampling strategy, drop to the low-level API. Use Config to choose the execution provider, Tokenizer to encode the prompt, and a Generator loop to produce tokens:

C#
// modelPath: the model folder; formattedPrompt: the prompt with the chat template applied.
using var config = new Config(modelPath);
config.ClearProviders();
config.AppendProvider("cuda"); // or "dml", "qnn"; leave empty for CPU

using var model = new Model(config);
using var tokenizer = new Tokenizer(model);
using var stream = tokenizer.CreateStream();

using var generatorParams = new GeneratorParams(model);
generatorParams.SetSearchOption("max_length", 2048);

using var generator = new Generator(model, generatorParams);
generator.AppendTokenSequences(tokenizer.Encode(formattedPrompt));

while (!generator.IsDone())
{
    generator.GenerateNextToken();
    Console.Write(stream.Decode(generator.GetNextTokens()[0]));
}

Use the package that matches your hardware: Microsoft.ML.OnnxRuntimeGenAI for CPU, or its .Cuda and .DirectML variants. For non-LLM models, a plain InferenceSession is enough, and the ML.NET guide shows how to score ONNX models inside ML.NET pipelines.

Hardware Acceleration: CUDA, DirectML, NPUs and Windows ML#

ONNX Runtime reaches hardware through execution providers (EPs). Each EP comes in a separate package or build, and a session can list several in priority order:

Execution providerHardwareTypical use
CPU (default)Any x64 or Arm64 CPUUniversal fallback, small models
CUDA, TensorRT RTXNVIDIA GPUsHighest throughput on workstations and servers
DirectMLAny DirectX 12 GPU on WindowsBroad Windows GPU support (now in sustained engineering)
QNNQualcomm NPUs, such as Copilot+ PCs on SnapdragonPower-efficient on-device inference
OpenVINOIntel CPUs, GPUs and NPUsIntel-optimized inference
CoreMLApple siliconmacOS and iOS apps
WebGPUGPUs through the WebGPU APICross-platform GPU access

Two recent shifts matter for Windows developers. First, the ONNX Runtime documentation now states that DirectML is in sustained engineering and that new feature work has moved to Windows ML. Windows ML is the Windows-supported copy of ONNX Runtime, shipped through the Windows App SDK (Microsoft.WindowsAppSDK.ML). It uses the same ONNX Runtime APIs and can acquire vendor-specific NPU and GPU execution providers through Windows Update, instead of you bundling them. Hardware-optimized NPU and GPU providers require Windows 11 24H2 or later. Second, ONNX Runtime itself added automatic EP selection policies, such as preferring the NPU or maximizing efficiency, for EPs registered with the environment.

Explicit provider selection remains the most predictable option for server workloads:

C#
using Microsoft.ML.OnnxRuntime;
using Microsoft.ML.OnnxRuntime.Tensors;

using var options = new SessionOptions();
options.AppendExecutionProvider_CUDA(deviceId: 0); // requires Microsoft.ML.OnnxRuntime.Gpu
using var session = new InferenceSession("intent-classifier.onnx", options);

float[] features = Preprocess(request); // your feature extraction
var input = new DenseTensor<float>(features, [1, features.Length]);
using var results = session.Run(new List<NamedOnnxValue>
{
    NamedOnnxValue.CreateFromTensor("input", input),
});
float[] scores = results[0].AsEnumerable<float>().ToArray();

Quantization Basics#

Quantization stores weights with fewer bits. The arithmetic is simple and explains most hardware requirements: weights need roughly parameters × bytes per weight.

PrecisionBytes per weightWeights for a 4-billion-parameter modelNotes
FP324about 16 GBTraining precision, rarely used for local LLMs
FP16 or BF162about 8 GBCommon on GPUs with enough memory
INT81about 4 GBGood quality, wide hardware support
INT40.5about 2 GBThe default for laptops and NPUs

Add memory for the KV cache, which grows with context length and concurrent sessions, and for the runtime itself. Lower precision also speeds up decoding, because each token requires reading fewer bytes. The cost is some quality loss, which tends to show up first in reasoning, arithmetic and instruction following. Techniques such as AWQ and GPTQ reduce that loss compared to simple round-to-nearest (RTN) quantization.

In practice, you rarely quantize models yourself. Ollama models are published as quantized GGUF files, and Foundry Local picks a quantized variant per device. For ONNX, the ONNX Runtime GenAI model builder converts and quantizes Hugging Face checkpoints:

Bash
python -m onnxruntime_genai.models.builder -m <huggingface-model-id> \
  -o ./models/my-model-int4 -p int4 -e cpu

Performance Expectations#

No single number describes local performance, because it depends on the model size, quantization, hardware, context length and runtime. Reason about it instead:

  • Decode speed tracks memory bandwidth. Discrete GPUs with fast VRAM generate tokens much faster than laptop CPUs, and a smaller or more aggressively quantized model is faster on any device.
  • Long prompts cost prefill time. RAG pipelines that stuff thousands of tokens into a small model's context pay for it in time to first token.
  • NPUs optimize power, not peak speed. They excel at sustained, battery-friendly workloads such as background summarization or live captions.
  • Cold start is significant. Loading gigabytes of weights takes noticeable time, so load once, keep the model warm, and show progress during first-run downloads.
  • Concurrency is limited. A desktop model typically serves one user well. Serving many users from one GPU requires a server-grade runtime and careful batching.

Measure on your target hardware. Record time to first token, tokens per second and peak memory for representative prompts, and include low-end devices from your user base.

Windows AI APIs#

On Windows 11, the Windows App SDK exposes built-in AI features that need no model management at all. They include Phi Silica (an on-device SLM for text generation and summarization), text recognition (OCR), image description, super resolution, segmentation and object erase. On Copilot+ PCs, these run on the NPU, and support is expanding to some GPUs and CPUs. Phi Silica is a Limited Access Feature that requires an unlock token. Microsoft's documentation states that Phi Silica is being replaced by a new on-device model, Aion Instruct, which starts rolling out to Windows Insider devices in October 2026 and to retail devices in November 2026. Plan for that transition if you build on Phi Silica today.

C#
using Microsoft.Windows.AI;
using Microsoft.Windows.AI.Text;

if (LanguageModel.GetReadyState() == AIFeatureReadyState.NotReady)
{
    await LanguageModel.EnsureReadyAsync(); // may download the model on non-NPU devices
}

using LanguageModel model = await LanguageModel.CreateAsync();
var result = await model.GenerateResponseAsync("Summarize: the build failed on the ARM64 leg.");
Console.WriteLine(result.Text);

These APIs are the lightest option for WinUI and WPF apps on supported hardware. They are not portable, so keep a fallback path for other devices.

Best Practices#

  • Code against abstractions. Use IChatClient and IEmbeddingGenerator so you can swap local and hosted models, and route requests by sensitivity or complexity.
  • Evaluate before you commit. Test candidate models on your own prompts. Also test tool calling and JSON output explicitly, because small models vary most there.
  • Pin model versions. Record the exact model, quantization and runtime version, and re-evaluate before you upgrade any of them.
  • Keep prompts short and specific. Small models follow focused instructions with a few examples much better than long, open-ended ones.
  • Manage first-run UX. Ask for consent before multi-gigabyte downloads, show progress, and cache models in a predictable location.
  • Apply safety measures locally. Hosted content filters do not apply to local models, so add your own input and output checks where needed. The multimodal AI guide covers local speech and vision scenarios.

Common Pitfalls#

  • Assuming hosted-model quality. An SLM that works in a demo may fail on edge cases. Measure accuracy on realistic data.
  • Ignoring memory limits. Loading an FP16 model on an 8 GB machine, or allowing very long contexts, leads to swapping or crashes.
  • Mixing native packages. Referencing both the CPU and GPU ONNX Runtime packages, or mismatched GenAI and ONNX Runtime versions, causes load failures.
  • Building new work on DirectML. DirectML still works but no longer gets new features. On Windows, prefer Windows ML or Foundry Local.
  • Depending on deprecated wrappers. Replace Microsoft.Extensions.AI.Ollama with OllamaSharp.
  • Exposing local servers unintentionally. Keep Ollama and Foundry Local endpoints bound to localhost unless you add authentication.

Ollama vs Foundry Local vs ONNX Runtime GenAI vs Windows AI APIs#

OptionBest for.NET integrationHardwareDistribution model
OllamaDeveloper machines, prototypes, internal serversOllamaSharp implements IChatClient and IEmbeddingGeneratorCPU and GPU through llama.cppSeparate service the user installs
Foundry LocalShipping on-device AI inside your own appNative C# SDK, or OpenAI-compatible endpoint plus AsIChatClientAutomatic CPU, GPU or NPU selectionSmall runtime with a model catalog and downloads
ONNX Runtime GenAIFull control, custom or fine-tuned ONNX modelsOnnxRuntimeGenAIChatClient or the low-level generator APIAny EP: CPU, CUDA, DirectML, QNN, OpenVINO, WebGPUYou ship the model files and native packages
Windows AI APIsBuilt-in features in Windows 11 appsWinRT APIs in the Windows App SDKCopilot+ PC NPUs, expanding to GPUsBuilt into Windows, no model to ship

Frequently Asked Questions#

Can I run an LLM locally in a .NET application?#

Yes. The simplest path is Ollama with the OllamaSharp client, which implements IChatClient. To embed models inside your own app, use Foundry Local or ONNX Runtime GenAI, which provides an OnnxRuntimeGenAIChatClient. On Windows 11 devices with supported hardware, the Windows AI APIs offer a built-in language model.

Should I use Ollama or Foundry Local?#

Use Ollama for development machines and internal servers, where installing a separate service is acceptable and its large model library helps. Use Foundry Local when you ship an application to end users and need automatic hardware selection, managed downloads and in-process inference.

Do I need a GPU or NPU for local AI?#

No, small quantized models run on modern CPUs, but slowly for long outputs. GPUs give the best speed, and NPUs give the best power efficiency on laptops. Plan for the weakest hardware your users have, and measure there.

What is quantization and does it hurt quality?#

Quantization stores model weights with fewer bits, for example 4-bit integers instead of 16-bit floats. That cuts memory and increases speed. It usually costs a little quality, most visibly in reasoning tasks, so evaluate quantized models on your own prompts.

Is DirectML deprecated?#

DirectML is in sustained engineering: it is still supported, but new feature development has moved to Windows ML. For new Windows projects, use Windows ML, Foundry Local or ONNX Runtime with vendor execution providers.

Summary#

  • Local AI trades some model quality for privacy, cost control, latency and offline use. Hybrid routing gives you both.
  • Ollama with OllamaSharp is the fastest way to experiment, Foundry Local is designed for shipping on-device AI, and ONNX Runtime GenAI offers full control with an IChatClient implementation.
  • Execution providers connect ONNX Runtime to CUDA GPUs, NPUs and more, and on Windows, Windows ML replaces DirectML for new work.
  • Quantization determines memory needs and speed. Do the arithmetic, and measure on real hardware.

Further Reading#