Meta Runs Google TPU Inference in Production via New PyTorch Engines
New open-source SGLang and vLLM versions let Meta run PyTorch models natively on Google AI chips without rewriting code for JAX.
Key points
- Meta runs SGLang-TorchTPU in production using Google TPUs.
- New engines remove the need to rewrite PyTorch models for JAX.
- Amazon also open-sourced TorchNeuron for its Trainium chips.
Meta now operates Google’s tensor processing units in its production AI infrastructure, utilizing new open-source inference engines that support PyTorch natively. According to Tom’s Hardware, the company announced the availability of PyTorch-native versions of the SGLang and vLLM engines on Google’s developer blog. This shift allows Meta to serve AI workloads on Google hardware without migrating its software stack to Google’s JAX framework, which was previously required for TPU compatibility.
The risk
The primary operational change involves moving away from JAX-dependent inference engines. Previously, both SGLang and vLLM relied on Google’s JAX framework to execute on TPUs. The new TorchTPU versions enable these engines to run PyTorch models directly on the chips. Meta’s PyTorch-TPU lead Zongwei Zhou, who works at startup RadixArk, stated that SGLang-TorchTPU is currently running in production at Meta. He described the result as a mature, PyTorch-native serving stack developed through collaboration between Meta, Google, and the SGLang team.
View the original post on X
Who is exposed
This development specifically affects organizations using PyTorch for AI inference that consider Google TPUs as hardware options. Jana van Greunen, director of PyTorch engineering at Meta, said the company cannot be constrained by a single hardware architecture. She noted that TorchTPU enables Meta to deploy PyTorch workloads to Google TPUs. Google stated it co-developed TorchTPU with Meta’s PyTorch team. This addresses Meta's historical reluctance to migrate its existing models to the JAX framework required by earlier TPU integration methods.
Cost and compliance
It remains unclear exactly where Meta’s production TPUs are physically located. However, Tom’s Hardware reported that discussions last year included potential Google Cloud TPU rentals in 2026 and hardware purchases in 2027. Meta reportedly later signed a multi-year rental agreement for these resources. This follows previous attempts by Meta research teams to use TPUs via PyTorch/XLA, an older bridge, between 2020 and 2023 before those contracts were canceled.
What should you do?
Teams should verify if their current inference engines support native PyTorch execution on target hardware to avoid framework migration costs. Engineers should test the new SGLang and vLLM TorchTPU versions in staging environments before deploying to production. Organizations relying on single-vendor hardware should assess whether multi-engine support reduces long-term infrastructure lock-in risks.
The background in plain language
AI inference engines translate trained models into live predictions. Frameworks like PyTorch or JAX provide the software language developers use to build these models. Historically, specific hardware often required models to be written in a matching framework, creating vendor lock-in. Native engines allow models built in one framework to run directly on different hardware without rewriting code. Teams usually evaluate inference performance and portability when selecting hardware. Check whether your chosen engine supports your preferred framework natively to simplify deployment pipelines.
What to do and how to stay safe: Google TPUs
- Review current inference engine compatibility with your preferred AI framework and hardware targets.
- Monitor vendor announcements for native framework support to reduce code migration overhead.
- Evaluate multi-vendor hardware strategies to mitigate supply chain or software lock-in risks.
- Test new open-source engine versions in isolated environments before production deployment.
Step-by-step guide: AI Governance Explained: Building Rules for Autonomous Systems
General security guidance from the Vector Tribune newsroom. It is not confirmed advice from the organisations named in this story.
Frequently asked questions
Does Meta use Google TPUs in production?
Yes, Meta runs SGLang-TorchTPU in production on Google TPUs using a new PyTorch-native engine.
Why did Meta move away from JAX for TPUs?
Meta prefers PyTorch and did not want to rewrite its models for Google’s JAX framework, according to Meta engineering leaders.
What other companies are releasing similar tools?
Amazon open-sourced TorchNeuron, a native PyTorch backend for its Trainium2 and Trainium3 chips, at the same conference.
