Google's New AI Chip Software Could Help It Take On NVIDIA

Deep News11-18 17:21

A software release last month went largely unnoticed by the general public, but it could have a major impact on Google's booming AI chip business.

At the end of last month, Google launched an open-source version of TorchTPU. This is a software middleware layer for Tensor Processing Units (TPUs), and it represents the biggest software-level step Google's chip business has taken in years.

The software overhaul is designed to lower the barrier for AI developers to migrate, allowing workflows originally built for NVIDIA (NVDA) graphics processing units (GPUs) to be moved onto Google TPUs.

From a broader perspective, Google hopes this software will replicate what CUDA has achieved for NVIDIA (NVDA). CUDA is the mainstream software layer bundled with NVIDIA (NVDA) AI chips and has long been one of the company's most important competitive moats.

Google's release of TorchTPU comes as more and more companies are looking for alternatives to NVIDIA (NVDA) chips, with NVIDIA (NVDA) having long dominated the AI chip market.

Reporting by me and my colleague Katherine Perlov notes that NVIDIA (NVDA) is mired in turmoil: founder and CEO Jensen Huang has departed, and its chip export business is also facing scrutiny from federal agencies.

Against this backdrop, the AI chip businesses of Google, Advanced Micro Devices (AMD), and Amazon are all expected to benefit.

AI developers have long favored the PyTorch artificial intelligence framework developed by Meta, which has been around for nearly a decade. PyTorch has strong user stickiness, but Google's own TPU framework, JAX, is cumbersome to use, and developers often feel that the effort to adapt to it is not worthwhile.

TorchTPU, jointly developed by Google and Meta, effectively bridges this gap. In an October blog post, Google said developers could run existing PyTorch workflows on TPUs with "minimal code changes."

"Now you can use the familiar tools to complete the entire process of model training, inference and deployment on TPUs," said Thomas Kurian, CEO of Google Cloud, which oversees the TPU business, on Alphabet's October 28 earnings call. He said the software update "opens up a much broader market" for TPUs.

Vadim Solovyov, CEO of cloud software company DoiT, told The Information: "The biggest obstacle in the past was 'you have to rewrite code with JAX.' In theory, since last month, that problem no longer exists."

Google TPUs began as an internal-only product more than a decade ago and have now grown into a product that can go head-to-head with NVIDIA (NVDA) GPUs.

Google previously mainly provided TPU compute to customers such as Anthropic and Apple in its own data centers; it has now begun selling chip hardware directly to smaller independent AI cloud providers such as Fluidstack.

Last month, Oracle (ORCL) announced it would deploy Google TPUs on its own cloud platform, making it the first major cloud service provider besides Google to support TPUs.

The Information previously reported that both Oracle (ORCL) and OpenAI had reached agreements to purchase TPUs.

TPUs are expected to add strong momentum to Google's already fast-growing cloud business, which doubled revenue last quarter to $30.7 billion.

A Morgan Stanley forecast in October showed that TPU hardware sales alone could exceed $81 billion by next year.

Of course, there is still huge room for growth. Solovyov's company tracks various cloud spending, and he estimates that among its roughly 5,000 customers, only 16 are actually using TPUs at scale.

The launch of TorchTPU also reflects a bigger crisis facing NVIDIA (NVDA): AI code generation capabilities are eroding the barriers built by CUDA, greatly accelerating the development and iteration of supporting software.

Anthropic co-founder and chief compute officer Tom Brown told investors that the company's Claude Mythos model shortened the cycle for adapting to Advanced Micro Devices (AMD) GPUs from as long as 12 months to just 3 days, and can generate almost all kernel code autonomously, weakening NVIDIA (NVDA) CUDA's moat.

Google is still continuing to step up its chip business. Earlier this month, Google released two new TPU chips, one of which was jointly developed by Google and Advanced Micro Devices (AMD). The company said it is more powerful and more secure than previous generations.

The turmoil hitting industry leader NVIDIA (NVDA) also brings opportunities for TPUs, potentially allowing them to break into the market of a group of emerging cloud providers (neoclouds) that previously relied heavily on NVIDIA (NVDA).

An executive at a neocloud told The Information that his company is in talks with Google about chip purchases, partly because demand for inference compute continues to rise — inference refers to running AI models that have already been trained, and Google claims its TPUs are more efficient in inference scenarios.

"We are customer-demand driven... our customers want to use TPUs. The entire industry is reaching a turning point where other companies' chips have become practical," the executive said.

Disclaimer: Investing carries risk. This is not financial advice. The above content should not be regarded as an offer, recommendation, or solicitation on acquiring or disposing of any financial products, any associated discussions, comments, or posts by author or other users should not be considered as such either. It is solely for general information purpose only, which does not consider your own investment objectives, financial situations or needs. TTM assumes no responsibility or warranty for the accuracy and completeness of the information, investors should do their own research and may seek professional advice before investing.

Comments

We need your insight to fill this gap
Leave a comment