Microsoft targets idle GPUs with dynamic composable memory for AI data centers

Microsoft is scouting system-level technologies that pool memory across servers to address the scarcity of memory resources in artificial intelligence data centers, according to Samir Rajadnya, principal architect at Microsoft.

The risk

Memory has become the most constrained resource in AI infrastructure, causing graphics processing units to remain idle while waiting for data, according to SiliconANGLE. This inefficiency creates significant operational waste for cloud providers managing large-scale inference workloads. Rajadnya stated that stranding memory behind unused workloads wastes a valuable resource during a period of tight supply and high demand.

Who is exposed

Hyperscalers and enterprises running AI inference workloads are directly affected by these supply constraints. Azure customers, including Delta Air Lines Inc. and Walmart Inc., rely on the platform as a utility, meaning any architectural changes must maintain strict reliability and security standards, according to Rajadnya. The pressure is visible across AI factories where compute resources are underutilized due to memory bottlenecks.

Cost and compliance

Idle capacity represents a substantial cost for cloud operators running at Azure’s scale. Rajadnya noted that AI inference is limited by both power and memory, creating two simultaneous ceilings for performance. The shortage of memory, widely reported in media, forces providers to use resources more efficiently to meet the demands of companies treating cloud services as critical utilities.

What should you do?

Teams should audit their current AI workloads to identify instances where GPUs are idle due to memory constraints. Organizations should evaluate whether their infrastructure allows for dynamic resource assignment or if they are locked into static server configurations. Monitoring power and memory limits will help prioritize which workloads benefit most from composable architecture updates.

The background in plain language

Composable infrastructure separates hardware resources like memory and storage from the processors that use them. Instead of buying a server with fixed memory, teams pool these resources in a central bank. Workloads request only the memory they need at that moment. This approach prevents waste when some servers are over-provisioned while others are starved. Teams usually check whether their current architecture allows this dynamic assignment or if hardware is permanently locked to specific processors.

What to do and how to stay safe: Microsoft

• Review current server configurations to identify static memory allocations that cannot be shared across workloads.
• Monitor GPU utilization rates to detect periods where processing power is idle due to memory bottlenecks.
• Evaluate vendor roadmaps for composable memory support to plan future infrastructure upgrades.
• Ensure security protocols account for shared memory pools where data may move between different tenant environments.

Step-by-step guide: AI Governance Explained: Building Rules for Autonomous Systems

General security guidance from the Vector Tribune newsroom. It is not confirmed advice from the organisations named in this story.

Frequently asked questions

What is composable memory?

Composable memory pools capacity beyond individual servers and assigns it to workloads on demand, rather than locking it behind a specific CPU or GPU.

Why is Microsoft changing its approach to memory?

Memory has become the scarcest resource in AI data centers, and Microsoft aims to use it more efficiently to avoid wasting capacity on idle servers.

How does this affect Azure customers?

Azure functions as a utility for companies like Delta Air Lines and Walmart, so new approaches must remain reliable and secure while improving resource efficiency.