99% Fewer Interruptions: Meta’s 1M-GPU Prometheus Supercluster Boosts AI Training Reliability

Meta has completed the first phase of its Prometheus supercluster in Ohio, a facility designed to train its Muse family of AI models. According to an engineering blog post published on Monday, the site currently operates hundreds of thousands of Nvidia GPUs and is expanding toward a full scale of approximately one million units. This infrastructure update follows earlier reports on Meta’s Muse Spark open model for Apple Silicon, as Vector Tribune noted in October 2026.

The risk

At this massive scale, hardware faults pose a significant operational risk. Meta reported that during the first phase in mid-2025, the cluster experienced nearly 1,000 interruptions per 8,000 GPUs each day. A single machine failure could halt an entire training run, creating substantial inefficiencies. The primary early issues involved signal quality across more than 600,000 NVLink connections and memory faults as the company scaled up newer GB300 racks.

Who is exposed

The Prometheus cluster spans multiple building types in New Albany, Ohio, including traditional data centers, AI-optimized structures, and rapid-deployment facilities. These buildings are linked by miles of fiber optic cable. Each individual building’s network connects up to 20,736 GPUs, while a separate aggregation network joins the buildings over distances ranging from 2 kilometers to more than 100 kilometers. Meta also utilizes capacity leased from external partners to support this distributed architecture.

Cost and compliance

Reliability improvements have come at a hardware cost. Each 72-GPU rack operates a 64-GPU slice while keeping eight GPUs as spares. Meta maintains approximately 2 percent of racks in reserve. Consequently, a training run using 32,768 GPUs actually requires 37,656 units, representing a 13 percent overhead. The cluster’s power draw swings by 30 percent to 40 percent as tens of thousands of GPUs compute in step, shifting tens of megawatts in seconds.

What should you do?

Managers overseeing large-scale compute resources should monitor for similar signal integrity issues when scaling GPU clusters. Teams should evaluate spare capacity requirements to mitigate the impact of single-point hardware failures. Organizations should also review their power management strategies to handle rapid load shifts associated with high-performance computing workloads.

The background in plain language

Training advanced AI models requires thousands of processors working in perfect synchronization. If one component fails, the entire process often pauses, wasting significant time and energy. Engineers must balance raw computing power with reliability, using spare hardware and sophisticated network management to keep systems running smoothly. This involves coordinating power delivery, data transmission, and hardware health across vast physical distances to ensure continuous operation.

What to do and how to stay safe: Meta

• Audit your current GPU cluster’s failure rates and compare them against industry benchmarks for large-scale deployments.
• Review your spare hardware allocation strategy to ensure you have sufficient redundancy for critical training workloads.
• Monitor network signal quality and memory health metrics proactively to identify potential bottlenecks before they cause outages.
• Evaluate your power management systems to handle rapid load changes without triggering grid-level constraints or instability.

Step-by-step guide: Browser security settings are the architectural walls of your digital life

General security guidance from the Vector Tribune newsroom. It is not confirmed advice from the organisations named in this story.

Frequently asked questions

How many GPUs will the Prometheus cluster have at full scale?

At full scale, Prometheus will comprise roughly 1 million distributed GPUs, according to Meta’s engineering blog.

What specific Nvidia platforms does the cluster use?

The cluster currently runs Catalina using GB200 NVL72 and Clemente using GB300 NVL72, with a future San Miguel platform planned for Vera Rubin NVL72.

How has Meta addressed the power fluctuations caused by the cluster?

Meta uses software called Dynamo Dimmer to cap GPU power gradually, supported by batteries at the rack, facility, and grid levels.