Microsoft Maia 300 AI Chip Uses Chiplet Design and HBM4 for 40% More Efficiency

Microsoft has officially unveiled the Azure Maia 300, a next-generation custom silicon accelerator designed for internal and partner artificial intelligence workloads. According to Neowin, the chip is built to operate at multi-gigawatt scale. This release marks a significant expansion of Microsoft’s hardware portfolio, aiming to reduce reliance on external suppliers for high-volume inference tasks. The company states that the Maia series will gradually become its primary source of inference capacity.

The risk

The primary challenge addressed by this hardware is energy efficiency in AI processing. SiliconANGLE reports that Microsoft estimates the Maia 300 can process 40% more tokens per megawatt than leading GPU-based systems, which likely refers to products from Nvidia Corp. This efficiency gain is critical as data centers face increasing pressure to manage power consumption while scaling AI services. The chip’s design directly targets the energy costs associated with running large language models at scale.

Who is exposed

The Maia 300 is optimized specifically for AI inference tasks. It features four computing chiplets manufactured using Taiwan Semiconductor Manufacturing Co.’s latest two-nanometer node. These chiplets are paired with eight HBM4 modules, providing a combined 288 gigabytes of memory capacity. This memory amount matches that of Nvidia’s flagship Rubin accelerator. However, the Maia 300 offers approximately 20% less memory bandwidth than the Rubin chip, a trade-off that Microsoft addresses through architectural changes.

Cost and compliance

Microsoft employs a dataflow architecture to mitigate the lower memory bandwidth. This design minimizes the constant movement of data between computing and memory circuits, which typically slows down processing. According to Microsoft, the chiplets keep information in an internal SRAM cache whenever possible, rather than sending it to the HBM stacks. When HBM usage is unavoidable, data is sent to physically nearby memory stacks, reducing transmission latency compared to distant stacks.

What should you do?

Microsoft executives Rani Borkar and Saurabh Dighe explained that the chip compresses data into lightweight formats like MXFP4. This compression reduces memory footprint and allows the system to support larger models and longer contexts within the same power envelope. The computing and memory modules are supported by a device control chiplet that manages power flow and collects telemetry. Two additional auxiliary chiplets handle data movement in and out of the accelerator.

What to do and how to stay safe: Microsoft

• Monitor vendor announcements regarding custom silicon adoption in cloud infrastructure to anticipate changes in service availability.
• Evaluate current AI workload power efficiency metrics to identify potential savings from newer accelerator architectures.
• Review data center cooling requirements, as Microsoft plans to deploy these chips in liquid-cooled racks.
• Assess memory bandwidth dependencies in your models to determine if lower bandwidth chips with optimized architectures are suitable.

Step-by-step guide: Mobile Security Architecture: 8 Practices for Device Integrity

General security guidance from the Vector Tribune newsroom. It is not confirmed advice from the organisations named in this story.

Frequently asked questions

How does the Maia 300 compare to Nvidia chips in terms of efficiency?

Microsoft estimates it processes 40% more tokens per megawatt than leading GPU-based systems, though it has less memory bandwidth.

What memory technology does the Maia 300 use?

It uses eight HBM4 modules with a combined capacity of 288 gigabytes, matching Nvidia’s Rubin accelerator.

How is the Maia 300 deployed in data centers?

Microsoft plans to deploy the chips in trays of four accelerators, placed in liquid-cooled racks containing 96 accelerators total.