Microsoft showed off the Maia 300 recently, the company’s third homegrown AI inference chip. Microsoft says it will pump out 40 percent more tokens per megawatt than the best GPU setups and is expected to ship by 2027. Power is the biggest bottleneck in AI data centers right now. Microsoft shaped the chip to tackle exactly that. Tokens are the bits of text a model takes in and spits out. Tokens per megawatt tells you how much work you get from each unit of power. Scott Guthrie, who heads Cloud + AI at Microsoft, put it this way: the company expects the chip to reach that 40 percent gain. Microsoft stayed quiet on which GPU it measured against and how it ran the tests. So the number comes from Microsoft itself. The company also says one rack of Maia 300 can push up to seven times the token throughput of a Maia 200 rack. The Maia 300 splits its work across several chiplets. TSMC builds the compute chiplets using its N2P process. Memory controllers and input/...
Microsoft showed off the Maia 300 recently, the company’s third homegrown AI inference chip. Microsoft says it will pump out 40 percent more tokens per megawatt than the best GPU setups and is expected to ship by 2027.
Power is the biggest bottleneck in AI data centers right now. Microsoft shaped the chip to tackle exactly that.
Tokens are the bits of text a model takes in and spits out. Tokens per megawatt tells you how much work you get from each unit of power. Scott Guthrie, who heads Cloud + AI at Microsoft, put it this way: the company expects the chip to reach that 40 percent gain. Microsoft stayed quiet on which GPU it measured against and how it ran the tests. So the number comes from Microsoft itself.
The company also says one rack of Maia 300 can push up to seven times the token throughput of a Maia 200 rack.
The Maia 300 splits its work across several chiplets. TSMC builds the compute chiplets using its N2P process. Memory controllers and input/output circuits sit on their own dies.
Every accelerator packs eight HBM4 stacks, 288GB of memory and 18 terabytes per second of peak bandwidth. It runs inside a 950-watt power envelope. Low-precision formats, including a beefed-up MXFP4, are supported, along with software-managed SRAM. These trim the memory needed for computation and for the key-value caches that pile up during long chats.
The AI chip goes after long-context reasoning, latency-sensitive conversations and multi-step agent tasks. Those jobs lean harder on memory and data movement than on pure compute.
A single Maia 300 rack carries 96 accelerators. It pulls 167 kilowatts and serves up 27.6TB of high-bandwidth memory with 1.7 petabytes per second of combined bandwidth. Ethernet ties the chips together. Stacked against Maia 200, Microsoft claims 2.5 times the memory bandwidth, over 2.5 times the performance per watt, roughly double the HBM bandwidth per watt, twice the accelerators per rack and 33 percent more memory capacity.
Google has TPUs. Amazon has Trainium. Microsoft went after both with Maia 200, saying it delivers three times the FP4 performance of Amazon’s third-generation Trainium. Everyone wants the same thing: cheaper AI serving and less reliance on NVIDIA, which still rules the market.
Reports say Microsoft has been haggling with TSMC for enough capacity to build more than 300,000 Maia 300 units for 2027. Those orders fight for the same advanced TSMC lines that NVIDIA, AMD, Google and Amazon all need. Microsoft also wants outside AI developers to pick up the chip.
This article, Microsoft’s new Maia 300 AI chip aims to cut its data center power costs, was originally published at NoypiGeeks | Philippines Technology News, Reviews and How to's.
Comments
Post a Comment