Big tech companies are cobbling together off-grid power systems to match their rushed timelines for AI development. What if these systems are glitchy?
Off-grid power comes with hefty price tags and operating risks. Tech giants are pursuing them anyway. Power providers, some of whom don't have much data-center experience, are jumping at the chance to sign deals with deep-pocketed customers. But cracks have shown at the few on-site power projects that have started up, highlighting potential costs to both tech companies and their power providers.
In the U.S., there are four operational data centers with off-grid or partially grid-connected power, according to data provider Cleanview. These are xAI's Colossus and Colossus II, Crusoe's Stargate Abilene and Vantage Data Centers' Vantage VA 2. Three of these have already had reported issues.
Earlier this month, Bloomberg reported that some data centers experienced cranks breaking off on combustion engines. The report also said that cracks had formed at gas-fired turbines powering xAI's Colossus data center in Memphis, Tenn.
In response to the report, Solaris Energy Infrastructure, which operates turbines for the campus, didn't directly confirm or deny the existence of turbine cracks on its earnings call but said its turbines were "in great shape." In an emailed statement, Solaris co-CEO Bill Zartler said the company has "not seen any extraordinary operating issues or failure rates on our generation equipment."
Cracked turbines and broken engine cranks both are major failures that require replacement, according to Nina Sadighi, founder of Eradeh Power Consulting, an electrical engineer who formerly worked at Amazon and Tesla.
In July, Vantage Data Centers experienced a critical power failure at its data-center campus in Loudoun County, Va., forcing the operator to switch over to diesel backup power for about a day, a spokesperson confirmed. Distributed Power Solutions, which Kodiak Gas Services bought earlier this year, operates the on-site power. Crusoe has also run into turbine glitches and other technical problems at Stargate Abilene, according to the Information. Stargate partner Oracle runs servers for OpenAI.
What's causing these problems? First, AI data centers' power demands can be incredibly taxing on power systems. AI workloads cause massive power demand spikes during computation, and a drop off as those computation results are gathered, according to a paper from Schneider Electric. As data centers get bigger, oscillations could reach hundreds of megawatts or even gigawatts, according to the paper. Big swings can stress and shorten the life of engines and turbines, and can even cause shaft fractures, according to the energy research and consulting firm Wood Mackenzie.
The same problem has surfaced on power grids, when grid-connected data centers suddenly drop power usage. Grid operators have had to scramble to take power supply offline to make sure the sudden excess of supply doesn't damage power plants and infrastructure.
Troy Patton, general manager at on-site power provider AlphaStruxure, said in an interview that he spoke to a potential hyperscaler client who required a very short power response time-tens of milliseconds, rather than hundreds of milliseconds as the industry is used to. These are response times that no rotating equipment, or a lithium battery, has ever had to respond to, he said.
Second, many of these off-grid power systems use a mishmash of different power equipment, depending on what was available to buy. It can be challenging to make sure all the different pieces work together.
Downtime can be costly for everyone involved. Anthropic, for example, is paying xAI a monthly fee of $1.25 billion for computing capacity on Colossus and Colossus II. While the contract details aren't public, the monthly fee implies that one day's worth of outage could be worth as much as tens of millions of dollars a day-either in lost fees for xAI or the cost to Anthropic of not having that computing power.
Outages can also be costly for power providers. These providers typically don't get paid for electricity they fail to provide, and in some cases must pay penalties for failing to deliver, according to industry experts. They are also typically on the hook for repairing and replacing broken power equipment. Sadighi said that at one of her client's former on-site data-center power systems, a power provider had to replace equipment worth millions of dollars following a multiday outage.
Power failures are the most acute risk, but a more insidious problem is if equipment keeps degrading more quickly than expected. Higher replacement costs would hurt power providers' margins.
Power providers do try to limit their maximum damages. Take Solaris Energy Infrastructure, the power provider to xAI's Colossus campuses. In its contracts, Solaris must refund the customer's fees when it fails to deliver power and must also cover the cost of repairing or replacing equipment, according to a public filing. But it can't be held liable for the customer's lost revenue from that downtime.
Of course, power providers build in redundancies to make up for these risks and model their best guesses for how long each piece of equipment will last. It is too early to tell if these are short-term learning pains or structural issues. "Nobody has enough operating history yet to say how this equipment holds up," Sadighi said. What is clear is that these data-center power loads are challenging enough to cause problems even for experienced, large-grid operators.
The stakes are getting higher because off-grid power systems are getting bigger. Amazon, for example, is planning an AI data-center campus in Pecos County, Texas, that would use natural-gas-powered turbines to generate as much as 7.65 gigawatts of power, according to Cleanview. That is equivalent to seven or eight nuclear power plants.
If these off-grid glitches are fixable learning pains, then power providers might only take a slight hit on their margins. But at worst, it could lead to losses, and a drying-up of data-center business if hyperscalers scrap their off-grid power plans. At least hyperscalers have highly profitable business models and strong balance sheets; many of these new power providers don't.
In addition to early players such as Solaris and VoltaGrid, many companies in the oil-and-gas industry-including Liberty Energy, Williams Companies, Atlas Energy Solutions and Kodiak Gas Services-have announced business units that will provide on-site power for data centers. Investors are willing to pay up for those with contracts. Solaris, which signed on two additional customers after xAI, fetches an enterprise value that is 10 times forward earnings before interest, taxes, depreciation and amortization, 19% higher than its oil-field service peer group.
Tech customer contracts are enticing indeed, but investors need to keep in mind that off-grid power at this scale is still an experiment. Technical challenges today can turn into financial problems in the future.
Comments