Microsoft Rolls Out Next Inference Accelerator to Boost AI in Azure

Microsoft recently announced the new Maia 200 AI inference processor. The company designed the processor for AI token generation for inferencing very large models, including reasoning and chain-of-thought models.

Microsoft Maia 200 AI inference processor.

Microsoft built Maia 200 around native FP8/FP4 tensor cores, complemented by 216 GB of HBM3e accessible at 7 TB/s. To keep the data models running, it has 272 MB of on-chip SRAM supported by data movement engines, a specialized DMA engine, and network-on-chip (NoC) fabric. These qualities reportedly result in Maia 200 delivering 30% better performance per dollar than Microsoft’s prior-generation AI silicon.

Faster Tokenization

Tokens are the lifeblood of most AI processing. AI uses matrix multiplication to condense multiple like patterns into tokens. Multimodal tokens represent text, images, sound, and video object components. These tokens are then compared with the input, which is assigned meaning based on the closest tokens.

Tokenization requires massively parallel processing. Early AI systems primarily utilized graphics processing units (GPUs) due to their large numbers of processing cores. However, large AI companies are now working toward specialized processors that are even more efficient at parallel math than GPUs. That’s where the Maia 200 comes in.

The Maia 200 delivers more than 10 petaFLOPS with FP4 and more than 5 petaFLOPS with FP8 math. AI often starts with lower precision calculations that use significantly less memory, power, and time. As tokenization progresses, higher-precision math is used for shorter periods. Native FP4 and FP8 can be used as is, combined in blocks, or operated on larger tensors, providing both economical processing and higher precision as needed.

Adding to Microsoft’s Heterogeneous AI Infrastructure

Microsoft intends to initially deploy the processor into Azure Cloud AI to speed Azure AI foundry and support Microsoft 365 Copilot. It is optimized for LLMs like OpenAI's and Microsoft’s first-party models under development. The chip also targets next-generation in-house models being developed by the Microsoft Superintelligence team.

Maia 200 server blade. 

Microsoft designed the chip for its own data center architecture. This bespoke design approach enabled faster turn-on, with AI models running just days after the arrival of first silicon. The implementation time from first silicon to rack deployment was half that of comparable prior compute infrastructure upgrade projects.

Multiple Efficiency Vectors

The Maia 200 can be scaled out over standard Ethernet to form clusters of up to 6,144 AI accelerators. The chip is built on TSMC’s 3-nm process node. It contains over 140 billion transistors. Maia 200 facilitates water-cooling, which is more efficient than air-cooling in data centers. The second-generation, closed-loop, liquid-cooling architecture is designed for zero water waste and intended to deliver the best token-per-watt-per-dollar processing. Maia 200 is already deployed and proving its worth in data centers in Des Moines, Iowa, and Phoenix, Arizona.

Comparison of Maia 200 with key competitors in AI processing.

Also available is the new Maia 200 software development kit (SDK). The SDK includes the open-source Python Triton compiler designed by OpenAI. It supports Meta AI's PyTorch, the open-source, Python-based deep learning framework. Along with that support, the SDK includes a Maia simulator and an AI cost calculator, enabling optimization early in the software development cycle.

All images used courtesy of Microsoft.