AI Chips & Infrastructure

Google Brings PyTorch Inference to TPUs as Meta Runs SGLang in Production

Google Developers BlogOct 26, 2026

Google introduced native PyTorch backends for vLLM and SGLang on TPUs, reporting that Meta is already using SGLang-TorchTPU in production.

Google announced native PyTorch backends for the vLLM and SGLang inference engines on its Tensor Processing Units, aiming to make TPU deployment easier for developers who already build and serve models with PyTorch. The launch includes TorchTPU, a backend that lets PyTorch run directly on TPU hardware, alongside TPU versions of both inference engines.

The announcement’s most significant deployment signal came from Meta. Google said SGLang-TorchTPU is running in production at Meta, and quoted Meta’s PyTorch-TPU lead describing the collaboration among Meta, Google and SGLang developer RadixArk. Meta’s statement said the stack enables PyTorch workloads to run on Google TPUs. The announcement did not disclose the scale of Meta’s deployment, the workloads involved, or whether the hardware is located in Meta facilities or accessed through a cloud arrangement.

The software is designed to reduce the need to adapt model code or serving infrastructure when moving PyTorch workloads to TPUs. Google described support for production features including tool calling, structured output, multimodal inputs, long-context serving and distributing inference across multiple hosts. Both engines are open-source projects, and Google said it built the TPU integrations with the teams behind vLLM and SGLang.

Google also reported benchmark results for its Ironwood TPU, including a cost-per-million-tokens figure and performance results for agentic workloads. Those figures are company-reported benchmarks, not an independent comparison across identical production configurations.

For the AI infrastructure market, the announcement shows Google extending TPU support into software ecosystems widely used by model developers. Meta’s reported production use is a concrete example of TPU deployment beyond Google’s own services, but the public announcement provides no information on volume, costs, or any commercial hardware agreement. The new backends may lower migration friction; their longer-term impact will depend on performance, availability and operational experience at larger scale.

Industry Impact

TPU serving becomes easier to evaluate

AI infrastructure, model serving, cloud computing

Overview

Google’s native PyTorch backends for vLLM and SGLang reduce the code and serving changes developers may need to move workloads onto TPUs. Meta’s reported production use of SGLang-TorchTPU provides a concrete adoption signal, though deployment scale, workloads, and hardware arrangements remain undisclosed. For ML engineers and software developers, the release broadens serving options within familiar tools. Infrastructure leaders should treat Google’s performance and cost figures as company-reported benchmarks, not independent proof of advantages across comparable production setups.

Key Impact Points

Native backends target lower migration friction for teams already using PyTorch and established inference engines.

Meta’s production use demonstrates deployment beyond Google’s own services, but its scale and commercial arrangement are unknown.

Production features include tool calling, structured output, multimodal inputs, long-context serving, and multi-host inference.

Opportunities

Evaluate PyTorch workloads on TPUs

AI infrastructure, model serving

Teams operating PyTorch-based inference services can assess whether the new vLLM and SGLang TPU backends reduce the work required to run existing workloads on TPU hardware. The available production features may also support evaluation of use cases involving long contexts, multimodal inputs, or tool calling. For infrastructure providers and internal platform teams, the practical opportunity is to test workload portability and serving operations before committing to broader adoption. Google’s benchmark figures are company-reported, and the announcement does not disclose Meta’s deployment scale, costs, or hardware arrangement, so teams should validate performance and economics against their own configurations.

Pilot PyTorch inference on TPU infrastructure
Problem: Moving PyTorch inference workloads to a different hardware platform can require adapting model code or serving infrastructure.
Solution: Evaluate Google’s TorchTPU backend with the TPU versions of vLLM or SGLang to test whether existing workloads can run with less adaptation.
Target Users: ML engineering and infrastructure teams serving PyTorch workloads.
Business Value: A pilot can establish workload compatibility and operational requirements before a wider deployment decision; the article does not quantify savings or performance gains.
Revenue Potential: No commercial pricing or revenue figures are disclosed. Any financial case depends on workload-specific testing and available hardware arrangements.

Reference link

https://developers.googleblog.com/announcing-new-vllm-and-sglang-inference-engines-on-tpu

Image Source Link

https://techcrunch.com/wp-content/uploads/2025/01/deepseek-2.jpg?resize=1200,800

Community Discussion (0)

Loading discussions...