Abstract:
Microsoft has launched RDMA over USB4 technology for Windows 11, which allows multiple ordinary PCs to work together through USB4 or Thunderbolt connections to share memory and GPU computing power, thereby running very large-scale AI models that originally exceeded the memory capacity of a single machine. This technology is expected to lower the hardware threshold for local AI inference and allow developers to use existing computers to build distributed computing systems.

This technology is based on Remote Direct Memory Access (RDMA). Unlike traditional network communications that rely on TCP/IP, RDMA allows connected devices to directly read and write each other's memory, bypassing some network processing links of the operating system and CPU, thereby reducing data copying, lowering communication delays, and improving transmission efficiency. RDMA was previously mainly used in high-performance servers and data centers, but Microsoft is introducing it to consumer Windows PCs this time.
Microsoft says RDMA over USB4 is implemented using the existing NetworkDirect interface and a new software RDMA provider. Compatible devices communicate directly via USB4 or supported Thunderbolt connections, eliminating the need for a dedicated RDMA network card or network switch. Tests show round-trip latency of approximately 28 to 50 microseconds when transmitting small messages, approximately four times lower than traditional Ethernet over USB4 communication based on UDP. This low latency is maintained with both 40 Gbps and 80 Gbps connections, although the latter has twice the nominal bandwidth.

In the actual demonstration, Microsoft collaborated with AMD to connect three mini PCs equipped with Ryzen AI Max+ 395 processor, Radeon 8060S graphics card and 128 GB unified memory through a fully interconnected USB4 40 Gbps network. Three devices successfully ran the Qwen3-235B-A22B hybrid expert model with 235 billion parameters, achieving an inference speed of approximately 18.5 tokens per second at 4-bit quantization. AMD's independent testing shows that the decoding speed can reach 21.98 tokens per second, and the prompt word processing speed can reach 117.98 tokens per second.


Intel also conducted a similar demonstration using three and four PCs with Core Ultra Series 3 processors, 64 GB of memory, and Thunderbolt 4 connections running the GPT-OSS-120B model. At a context length of 1000 tokens, the speed is about 49.4 tokens per second with three devices, rising to 54.2 tokens per second with four devices. It is worth noting that this 120 billion parameter model cannot be completely loaded into a single PC with 64 GB of memory, so multi-machine collaboration becomes the key to running the model.
Microsoft pointed out that RDMA over USB4 is still in the experimental stage and has high requirements for both hardware and software configuration. Participating PCs must have a Windows-managed USB4 connection, preferably at 40 Gbps or higher. Microsoft also recommends a full-mesh topology, where each PC is directly connected to every other device, rather than daisy-chaining, which can significantly reduce performance.

In addition, applications must explicitly support distributed execution through a compliant framework. Microsoft has open sourced the NetworkDirect provider and released a modified llama.cpp implementation demonstrating the technology. RDMA over USB4 is expected to be launched with a future "Future Platforms" version of the Windows Insider Experimental Channel, but Microsoft has not yet announced a specific release time.
During the developer preview phase, this feature is turned off by default and needs to be enabled through registry configuration and test signature; enabling test signature may also require temporarily turning off Secure Boot. Microsoft therefore recommends that users test on dedicated development equipment rather than computers they use every day.
Although the official timeframe for widespread adoption is uncertain, if this technology eventually comes to fruition, it could make powerful local AI inference more commonplace. Developers may no longer have to purchase expensive dedicated AI servers, but can combine the memory and GPU resources of multiple existing PCs to run larger-scale local AI models.
Comments