[Beowulf] oil immersion cooled blades

[Eugen Leitl]

http://www.heise.de/newsticker/meldung/Server-Blades-in-Oel-1471734.html

(translation courtesy Google Translate):

Server blades in oil

Hardcore Computer LSS 200

Photo: Boston The U.S. company has with the hardcore computer Submerged
Liquid server developed (LSS 200), a server blade in the format of immersion
cooling uses: The entire unit sits in a closed housing which oil flows. This
"Core Coolant" is according to the safety of a non-toxic and biodegradable
compound based on a synthetic wax. The advantages of this cooling method is
called Hardcore Computer eliminated including a more efficient cooling,
because the coolant can be for example of a data center transported directly
to heat exchangers, and going through cold air. Because no special rack
delivers warm air directly into the environment, it can also be operated in
locations without air conditioning.

The server manufacturer in Boston, the LSS 200 is added to its product line
and offers it in Germany. However, neither prices nor called Boston delivery,
and plugged in the server rack now slightly dusty Technology: In comparison
to the recently announced Xeon E5600 Xeon E5 falls off significantly. Also,
the data sheet (PDF file) of the CLS 200 with hardcore computer can open
questions. For example, detailed information is lacking on the power
supplies, the special chassis and to the oil-cooling equipment. The CLS 200
will also allow the use of a PCIe expansion card, such as a Tesla accelerator
or InfiniBand adapter card - if it works well both at the same time remains
uncertain. Finally, the question remains open on the disk, at least be
mentioned only SSDs in 2.5-inch format.

The entire board is surrounded by oil. Picture: Hardcore Computer

In the United States sold the hardcore computer desktops and Reactor Reactor
X, and the detonator Workstation with Immersion cooling. Oil as a coolant has
a lower specific heat capacity than water, but cools better than air and
leaks caused by short circuits. When complete immersion of the coolant
reaches all the components, while in the water cooling individual heat sinks
are needed which do not reach all critical components of any assemblies.
Optimal cooling water can be exploited if the motherboard design optimized to
be. ( ciw )

[Mark Hahn]

Post by Eugen Leitl
Server blades in oil

sounds like some brutalist take on molecular gastronomy!

Post by Eugen Leitl
Hardcore Computer LSS 200

http://www.hardcorecomputer.com/Resources/assets/Documents/LSS-specs.pdf

seems kind of uninspiring, in hardware specs. 1366 socket, not 2011,
optional fast network, gpu, etc, builtin ipmi. 8x in 5U, so not really
a density play.

I guess I'm a bit skeptical about the utility of this approach -
would be nice if they had some technical literature. something about
thermal resistance. define how the oil bath dumps the heat (water
hookups in the back?) comparison to modern heatpipe-based solutions, etc.

[Prentice Bisbal]

Post by Mark Hahn

Post by Eugen Leitl
Server blades in oil

sounds like some brutalist take on molecular gastronomy!

Post by Eugen Leitl
Hardcore Computer LSS 200

http://www.hardcorecomputer.com/Resources/assets/Documents/LSS-specs.pdf
seems kind of uninspiring, in hardware specs. 1366 socket, not 2011,
optional fast network, gpu, etc, builtin ipmi. 8x in 5U, so not really
a density play.
I guess I'm a bit skeptical about the utility of this approach -
would be nice if they had some technical literature. something about
thermal resistance. define how the oil bath dumps the heat (water
hookups in the back?) comparison to modern heatpipe-based solutions, etc.

No need for water hookups. You can circulate the oil through an oil to
air heat exchanger. That's what those green cooing guys at the SC
conferences do.

Prentice

[Mark Hahn]

Post by Prentice Bisbal

Post by Mark Hahn
I guess I'm a bit skeptical about the utility of this approach -
would be nice if they had some technical literature. something about
thermal resistance. define how the oil bath dumps the heat (water
hookups in the back?) comparison to modern heatpipe-based solutions, etc.

No need for water hookups. You can circulate the oil through an oil to
air heat exchanger. That's what those green cooing guys at the SC

is that really better than going to air directly?
I guess I'd like to see the the numbers - to my way of thinking,
it's almost all about the thermal resistance. transferring heat to oil,
then to air, means two stages of resistance. using oil would permit
a bigger air interface, though I suppose.

[Nathan Moore]

If you take a cm^3 of space, right next to the cpu and fill it either with
air, or with oil, you'll have many, many, more atomic/molecular degrees of
freedom to fill with energy in the cm^3 of oil. Getting that energy out of
the cooling medium seems primarily like a fluid-flow problem - given oil's
higher heat capacity, you can leave it around something hot, and still have
it serve as an effective heat sink for a longer period of time than you can
with air. My point is, the fan for the air has to run much faster than the
oil pump for the oil coolant.

I'm too young for this, but didn't VW and Porche cool some of their engines
with oil through the early 1980's?

Post by Mark Hahn

Post by Prentice Bisbal

Post by Mark Hahn
I guess I'm a bit skeptical about the utility of this approach -
would be nice if they had some technical literature. something about
thermal resistance. define how the oil bath dumps the heat (water
hookups in the back?) comparison to modern heatpipe-based solutions,

etc.

Post by Prentice Bisbal
No need for water hookups. You can circulate the oil through an oil to
air heat exchanger. That's what those green cooing guys at the SC

is that really better than going to air directly?
I guess I'd like to see the the numbers - to my way of thinking,
it's almost all about the thermal resistance. transferring heat to oil,
then to air, means two stages of resistance. using oil would permit
a bigger air interface, though I suppose.
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf

[Prentice Bisbal]

Post by Nathan Moore
If you take a cm^3 of space, right next to the cpu and fill it either
with air, or with oil, you'll have many, many, more atomic/molecular
degrees of freedom to fill with energy in the cm^3 of oil. Getting
that energy out of the cooling medium seems primarily like a
fluid-flow problem - given oil's higher heat capacity, you can leave
it around something hot, and still have it serve as an effective heat
sink for a longer period of time than you can with air. My point is,
the fan for the air has to run much faster than the oil pump for the
oil coolant.
I'm too young for this, but didn't VW and Porche cool some of their
engines with oil through the early 1980's?

Yes, I and was going to mention that in the e-mail. I just posted. The
original VW "boxer" engines were air cooled. If you look at the
cylinders, they have heat fins on them like a CPU heat sink or a
motorcycle engine to promote heat transfer to the air. For VW fanatics
(of which I used to be one) there two types of VWs: air-cooled, and
everything else. I was in the "everything else" camping, owning several
rabbits and GTis.

The engines in the early Porsches and the 911s trace their lineage all
the way back to the first VW beetle engines, and were also air-cooled
boxer engines (boxer = horizontally-opposed cylinders). Porsche 911s
retained air-cooled engines well into the 1990s, and went to
water-cooled engines because it was getting too hard to meet tougher and
tougher noise and emissions requirements with air-cooled engines. As a
bonus, I think liquid-cooling really helped increase how much horse
power they could put out, too. But you lose out on the very distinctive
sound that the air-cooled 911 had. :(

Do you remember the 911 turbos with that big "whale tale" rear wing"
That wing wasn't really there for aerodynamics (although it did help for
that, too) It was there to house the massive air-to-air intercoolers
needed to cool the turbocharged air. Since they went to liquid cooled
engines, you really don't see that any more.

Prentice

[Prentice Bisbal]

Post by Mark Hahn

Post by Prentice Bisbal

Post by Mark Hahn
I guess I'm a bit skeptical about the utility of this approach -
would be nice if they had some technical literature. something about
thermal resistance. define how the oil bath dumps the heat (water
hookups in the back?) comparison to modern heatpipe-based solutions, etc.

No need for water hookups. You can circulate the oil through an oil to
air heat exchanger. That's what those green cooing guys at the SC

is that really better than going to air directly?
I guess I'd like to see the the numbers - to my way of thinking,
it's almost all about the thermal resistance. transferring heat to oil,
then to air, means two stages of resistance. using oil would permit
a bigger air interface, though I suppose.

Oh boy... I'm having flashbacks of college. The short answer is "yes"
that is much better than going to air directly. I'm tying to think of a
simple way to explain why, and I'm starting to have flashbacks of my
Chem Eng. classes: Thermodynamics, Transport Phenomena... ugh.

Here's an attempt at a longer explanation. Sorry if I'm rehashing what
you already know.

Yes, going from oil, or any liquid, to air is better than using air
directly. That is why car engines are liquid-cooled instead of air
cooled. It comes down to two physical properties: Thermal conductivity,
and thermal capacity.

Thermal conductivity is the ability for a material to move energy in the
form of heat from one place to another. Thermal capacity is how much
energy in the form of heat can be stored in a given quantity (mass or
volume) of a substance.

In general, liquids and solids have better thermal conductivity and
capacity than gases. We use liquid for cooling because the liquid's
superior conductivity draws the heat away from the object to be cooled
quicker, requiring less surface area, and less "residence time" for the
coolant. The superior capacity allows us to use less volume of coolant
to store the same amount of "heat" (energy). And liquids are better than
solids because they flow, and the bulk movement allows us to easily
transport the heat stored in the coolant away easier than if were trying
to move solids.

Using liquids allows to do draw heat out of tiny confined spaces, like a
1U server, or a cars engine block, that we can't do effectively with
air. Once we draw that heat out, it needs to go somewhere, which, in
most cases, is atmospheric are, so we pump the liquids to a cooling
tower outside, the radiator in the front of our engine bay, or
somewhere where is a large amount of air and room for all the surface
area the heat exchanger will need (since air has a low thermal
conductivity, it needs a lot more surface area to transfer the heat into
it than a liquid does). This last point you hit on in the last sentence
of your post. That is why car radiators are large and have lots of
think aluminium fins in them (more fins = more surface area for the air
to contact), and we have massive cooling towers or heat exchangers
outside our office buildings.

You are correct in that there are inefficiencies when you transfer
energy from one material to another, but those are usually trivial and
offset by the greater effectiveness of liquids in pulling the heat away
from the heat source in the first place.


I hope that helps.

Prentice

[Lux, Jim (337C)]

Immersing motherboards in oil has been done for years (probably even
mentioned on this list). There's countless hacker versions using various
and sundry off the shelf mobos and power supplies etc in various and
sundry containers (ice chests, aquariums, purpose built acrylic boxes)
using a variety of coolants.



The vast majority of these have no "thermal engineering".. They're more
"we tried it and it seemed to work". There's not a lot of content at the
hardcore computer website. It could be ok or it might not. They're not
real clear on how they actually get the heat out of the enclosures. The
MSDS for their coolant says that it's basically synthetic mineral oil with
a antioxidant (you can look up the CAS number to get more info). I'd be a
bit nervous about the fire hazard aspects of a machine room full of the
stuff.

So here's my experience using oil as an insulator/coolant.
1) it wicks up insulated wire, particularly stranded. Put the mobo in oil
and the power supply outside, and pretty soon your power supply will be
full of oil.
2) oil leaks. There is *nothing* that is oil insulated that doesn't have
a fine film of oil on its surface eventually, unless it is in a
hermetically sealed can with welded/crimped seals.
3) oil is a mess when you need to fix something



The Cray-2 used Fluorinert(tm) FC-74 as the coolant, which is very nice to
work with, although expensive.
It doesn't wet things very well, so when you pull something out of the
bath, it doesn't bring much fluid with it. The Cray used it as a heat
transfer medium to water coolant. I think they had a way to drain it into
a tank quickly for servicing.


It can be used for ebullient (boiling) cooling by picking the right vapor
pressure/BP grade (the Cray didn't use this). Ebullient cooling is quite
efficient at moving the heat away because it's a phase change, and the
bubbling causes good circulation, but it does require careful design so
you don't get film boiling/Leidenfrost effect (the phenomenon that
protects your feet when walking across burning coals barefoot)

Post by Mark Hahn

Post by Eugen Leitl
Server blades in oil

sounds like some brutalist take on molecular gastronomy!

Post by Eugen Leitl
Hardcore Computer LSS 200

http://www.hardcorecomputer.com/Resources/assets/Documents/LSS-specs.pdf
seems kind of uninspiring, in hardware specs. 1366 socket, not 2011,
optional fast network, gpu, etc, builtin ipmi. 8x in 5U, so not really
a density play.
I guess I'm a bit skeptical about the utility of this approach -
would be nice if they had some technical literature. something about
thermal resistance. define how the oil bath dumps the heat (water
hookups in the back?) comparison to modern heatpipe-based solutions, etc.
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf

[Eugen Leitl]

Post by Lux, Jim (337C)
The Cray-2 used Fluorinert(tm) FC-74 as the coolant, which is very nice to
work with, although expensive.

I've worked with a few 3M fluorinerts for blood substitutes (nothing
kills quite like liberated fluoride during sonickating) and for cooling/partial
liquid ventilation purposes. Heavy stuff, twice the density of water.
Seems to do havoc to your immune system at long-term exposures.

Post by Lux, Jim (337C)
It doesn't wet things very well, so when you pull something out of the
bath, it doesn't bring much fluid with it. The Cray used it as a heat
transfer medium to water coolant. I think they had a way to drain it into
a tank quickly for servicing.
It can be used for ebullient (boiling) cooling by picking the right vapor
pressure/BP grade (the Cray didn't use this). Ebullient cooling is quite

The Cray X1 used evaporative spray cooling at least
http://www.cse.ohio-state.edu/~panda/875/class_slides/cray-jaguar.pdf

Post by Lux, Jim (337C)
efficient at moving the heat away because it's a phase change, and the
bubbling causes good circulation, but it does require careful design so
you don't get film boiling/Leidenfrost effect (the phenomenon that
protects your feet when walking across burning coals barefoot)

There's a K/Na eutectic which is cheap (unlike Ga eutectics) and
is liquid at RT. Extreme fire hazard if you'll get a leak in air, though.