Euro cloud biz trials ‘server blades in a cold box’ system • The Register

A1 Digital is testing liquid cooling tech for power-intensive AI servers that is claimed to save 50 percent of the energy previously required for cooling, doing away entirely with the need for air conditioning.

Exoscale, A1 Digital’s cloud and hosting provider, is running a proof of concept (POC) in its Vienna datacenter of a direct liquid cooling (DLC) system developed by Austrian company Diggers.

This puts server blades into a sealed cold box, with no air coming in or out, and cold plates with microchannels to circulate the liquid attached directly to the GPUs and CPUs. This setup, we’re told, eliminates the need for a cold aisle, hence claims that it can save 50 percent of the energy costs and deliver a PUE of about 1.05.

Diggers cold plate and GPU – click to enlarge

PUE, or power usage effectiveness, is a measure of how energy efficient datacenter infrastructure is, with the ideal being 1.0. The industry average is about 1.56, according to Uptime Institute.

Exoscale chief operating officer Antoine Coetsier told us that everything in the datacenter is getting denser and consuming more energy, while electricity is getting more expensive and companies like his are coming under pressure over sustainability, especially with the EU’s Corporate Sustainability Reporting Directive (CSRD) now in force.

“So every generation, not only GPUs, but also CPUs now, are consuming more energy. The current generation is 350 watts just for the CPUs. The things we are going to start deploying from the end of March are going to be 500 watts per CPU.”

This is what has driven the evolution of cooling from air conditioning the entire data hall, down to a cold aisle between racks of servers, and now this approach of liquid cooling inside a cold box enclosure, which is insulated like a fridge.

Diggers CEO Martin Schechtner says this latest technology represents its fourth generation of design.

Diggers cold box open – click to enlarge

“It is a cold box approach, meaning that there’s no air going in or out of the server rack. It is only water in, water out, power supply, and network to make it the easiest way,” he told us.

Previously, Diggers sold immersion-based cooling, but that typically uses mineral oil, which means “you are putting a lot of fire load into a highly critical infrastructure,” Schechtner said, as well as making maintenance trickier.

With DLC applied to server blades inside a cold box, “we basically achieve the same technical results as immersion cooling, but have no mineral oil and no maintenance issues. Everything is easy to change. You can swap memory and drives. There’s no oil dripping,” he explained.

Diggers’ cold plates are milled from aluminium rather than copper to avoid corrosion issues from water, and the company claims the microchannels distribute the liquid in a way that increases heat transfer. One cold plate is used to cool two GPUs, with one on each side of a plate, and on the POC, each cold box contains four blades mounted vertically.

“Everything is contained and there’s very few pipes. There’s still a few pipes there, but much fewer than with anything that you’ve seen before even from the big OEMs, which makes one of the benefits of the system as well,” Coetsier said.

The deployment being trialled at the Vienna datacenter comprises eight boxes in total, which is about 80 kilowatts of compute capacity.

“It is not huge because this is a POC. We’re going to ramp it up, but it has the potential to host the most powerful GPUs available,” Coetsier told us.

The water coming out of the cold box is typically at 50°C, according to Schechtner, “and this is a temperature level where you can either make direct usage on-premises or you could feed it into district heating networks where a heat pump would work very efficiently.”

Coetsier says that Exoscale intends to bring this cooling tech to other locations beyond its Vienna datacenter, assuming the POC goes well, and to other mixes of CPU and GPUs combinations “because we are a cloud provider, so our job is to provide whatever the customers need and want to buy as a service.” ®