Microsoft has updated the CUDA plugin for ONNX Runtime, introducing a feature that lets certain Mixture of Experts models run even when they don’t fit entirely in GPU memory. The CUDA plugin 0.4.0, released on November 14, enables offloading less-used model experts to the CPU while keeping the ones needed for active inference on the GPU. The feature requires ONNX Runtime 1.32.0.
How CPU-GPU Offload Works in MoE Models
Mixture of Experts models consist of multiple specialized sub-networks, each called an expert. For any given input, only a subset of those experts gets activated. The new expert offload feature exploits exactly this structure: experts can remain stored in system memory and transfer to the GPU only when needed, cutting the VRAM required to hold the full model on-device.
ONNX Runtime offers an automatic mode that keeps experts on the GPU when space allows and switches to hybrid execution when memory runs short. A mandatory mode is also available, which causes session creation to fail if the model can’t be handled within the configured requirements. Both modes are configured through session options, with no changes to the model itself required.
Less VRAM, But Latency Must Be Benchmarked
The update widens the range of hardware configurations capable of running compatible MoE models. That said, it doesn’t mean every model will run on any GPU: support covers specific nodes and formats, including QMoE and compatible quantized models.
The performance trade-off also needs to be verified case by case. Shifting part of the computation to the CPU can relieve pressure on graphics memory, but the release notes don’t specify a single latency penalty figure. The actual result will depend on the model, quantization settings, CPU, system memory, and GPU in use.
Version Compatibility Requirements
The CUDA plugin 0.4.0 is paired with ONNX Runtime 1.32.0. Certain features, especially MoE offload, depend on new capabilities introduced in the core runtime. Microsoft recommends aligning versions and notes that operator schema differences can cause incorrect execution or crashes.
The release also includes support for GGUF and block-packed weights, quantized kernels, and optimizations for long-context inference and speculative decoding. For developers and deployers working with large models, the central takeaway is the option to use CPU and GPU together when video memory is the bottleneck. Real-world advantage in throughput and latency will still need to be measured with benchmarks on actual hardware and models.
Sources: ONNX Runtime CUDA Plugin EP 0.4.0 · ONNX Runtime 1.32.0
