AI requirements exceed infrastructure capabilities for many IT teams, study finds

As adoption of artificial intelligence (AI) technology accelerates, IT organizations are concerned that their existing infrastructure isn’t powerful enough to keep up.

AI hardware – especially training hardware – is becoming more and more power hungry, according to Equinix, which just released its 2023 Global Tech Trends Survey.

[ Download our editors’ PDF AI as a service (AIaaS) enterprise buyer’s guide today! ]

The power draw from traditional racks in a data center is between 5 kW and 10 kW per rack. But, increasingly, newer generations of GPU-based racks are pushing power draws north of 30 kW per rack, and in some cases as high as 72 kW per rack, according to Kaladhar Voruganti, senior technologist at Equinix. “So, definitely, it’s very hard to host this type of infrastructure in private data centers,” he said.

In its research, Equinix found that 85% of the 2,900 IT decision-makers polled are already using AI or planning to use it across multiple key functions. Organizations are most likely to be using AI, or planning to do so, in IT operations (85%), cybersecurity (81%), and customer experience (79%).

However, 42% of IT leaders surveyed believe their IT infrastructure is not fully prepared to meet the demands of AI technology.

Power capacity isn’t the only concern. Cooling, too, is impacted by AI’s climbing power demands.

Traditional air cooling involving heatsinks and fans only works effectively until roughly 30 kW per rack. After that, the fans simply can’t keep up, and liquid cooling is required. Most data centers are not equipped for liquid cooling and would have to be retrofitted at considerable expense.

Along with hardware shortcomings, many companies are dealing with a shortage of skilled IT professionals. There has always been a shortage of people with analytics skills, for example, and that has only been exacerbated with the explosion in popularity and use of AI.

AI as a service

Companies lacking in the proper hardware to do AI training have two options: make a massive investment in hardware or turn to cloud service providers for AI-as-a-service, which most of the top cloud service providers now offer. Rather than make the million-dollar investment in hardware, an enterprise could upload the data to be processed and the cloud service provider can do the heavy lifting. The enterprise could take the trained data models back when the processing is done.

Customers often will opt for end-to-end solutions from AI vendors in the cloud, especially initially, “because they make it easy for the customers with a simple button,” Voruganti said. But variable cloud costs – which enterprises incur with each read or write to cloud-based data, or with every data extraction, for example – may cause IT teams to reconsider that approach.

Voruganti said he’s seeing companies choose to place foundation models with different cloud service providers based on their areas of expertise. So, one provider might offer strong vision-based models, while another might have better general language or large language models.

“There is increasingly desire for people to leverage these models from different clouds,” Voruganti said. “And [if they can maintain their own control over their data], like a neutral location, then they can bring the model to where the data is, and then customize it.”