Discussion:
MS Cray
(too old to reply)
Gus Correa
2008-09-16 20:20:50 UTC
Permalink
Dear Beowulf and COTS fans

For those of you who haven't read the news today:

http://www.theregister.co.uk/2008/09/16/cray_baby_super/

IGIDH (I guess it doesn't help.)

Gus Correa
--
---------------------------------------------------------------------
Gustavo J. Ponce Correa, PhD - Email: ***@ldeo.columbia.edu
Lamont-Doherty Earth Observatory - Columbia University
P.O. Box 1000 [61 Route 9W] - Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------
Prentice Bisbal
2008-09-16 20:35:31 UTC
Permalink
Post by Gus Correa
Dear Beowulf and COTS fans
http://www.theregister.co.uk/2008/09/16/cray_baby_super/
IGIDH (I guess it doesn't help.)
Gus Correa
Quote from article:

"It's also attempting to lure scientists and researchers with
discretionary IT budgets to forget using shared, giant clusters and get
their own box and tuck it in behind their desk where no one can see it
to run their workloads locally. The personal supercomputer is not a new
idea, but this is the first time that Cray is trying it out in the market."

That will work great until the newbie scientists find that airflow into
a computer tucked in "behind their desk where no one can see it" is piss
poor, and that fans powerful enough to provide adequate airflow "behind
the desk where no one can see it" are going to be LOUD.
--
Prentice
Prentice Bisbal
2008-09-16 21:01:41 UTC
Permalink
Post by Prentice Bisbal
That will work great until the newbie scientists find that airflow into
a computer tucked in "behind their desk where no one can see it" is piss
poor, and that fans powerful enough to provide adequate airflow "behind
the desk where no one can see it" are going to be LOUD.
Ahem.
"Because the CX1 sits in an office environment, the front of the chassis
has an optional noise cancellation add-on, which drops the whirring of
fan noise down to the point where it is actually legal to put it in an
office environment."
When I worked at Streamline, we demoed a blade cluster in a
noise-cancelling APC rack enclosure at a conference we ran. It sat
running away at full lilt at the back of the room as people delivered
their talks, and no-one noticed till we opened the front door.
I guess this is somethign similar.
You got me. I saw that when I continued reading the article *after* my
post. I was hoping no one else read the article to the end.

Noise-cancellation devices may help keep the noise down, but the air
flow under or "behind" a desk is still a problem. Fans can only move air
if there's a place for the air to come from, and a place for the air to go.
--
Prentice
John Hearns
2008-09-16 21:23:21 UTC
Permalink
Post by Prentice Bisbal
You got me. I saw that when I continued reading the article *after* my
post. I was hoping no one else read the article to the end.
Noise-cancellation devices may help keep the noise down, but the air
flow under or "behind" a desk is still a problem. Fans can only move air
if there's a place for the air to come from, and a place for the air to go.
I do agree with you there. Many times we've seen personal supercomputers
touted, only for them to fade away. Fancy enclosure or no, you have to
think twice about one of these under your desk. Networks are fast these
days.

I've also seen plenty of people who buy quite ordinary perforated steel
racks and are convinced a cluster will sit happily in a work area, or in a
room dedicated to the cluster plus graphics workstations (I can think of
examples from Streamline and from my company previous to that). Such users
tend to rapidly abandon that idea!

I once got an Intel twin motherboard system to take home, for evaluation and
power draw measurements for a contract at CERN. I live in a two bedroom
apartment in central London. I couldn't power this thing up for fear of the
neighbours being woken up, though I'm quite happy to share a machine room
with 200 of the things.

John Hearns
Alan Louis Scheinine
2008-09-16 22:13:16 UTC
Permalink
It can be viewed as a seasonal machine.
To save energy in winter some institutions lower the
thermostat so that wearing a sweater is necessary.
Lucky is the parallel programmer with a mini-cray under
the desk.

A personal Cray does not simplify life for many people.
I heard someone say that the disadvantage of working as
a private consultant is the time spent being one's own
IT department. For many people, even in technical fields,
maintaining MS Windows (not to mention Linux) is difficult
or at least annoying. So even a simplified Cray with MS
Windows at a company would probably not be "personal" but
rather maintained by the IT support. I welcome comments
from other people as to whether I'm accurate about my sense
of what is typical at company.

The key added value (aside from the Cray nameplate) is
the support for easy installation. But I remember the
Cray XD1 (acquired from Octiga Bay). If sales volume
is low the product line might be dropped so the specialized
simplifications provided by Cray may not be useful after
a few years.

Best regards,
Alan Scheinine
--
Alan Scheinine
5010 Mancuso Lane, Apt. 621
Baton Rouge, LA 70809

Email: ***@tuffmail.us
Office phone: 225 578 0294
Mobile phone USA: 225 288 4176 [+1 225 288 4176]
Lux, James P
2008-09-16 22:07:47 UTC
Permalink
James Lux, P.E.
Task Manager, SOMD Software Defined Radios
Flight Communications Systems Section
Jet Propulsion Laboratory
4800 Oak Grove Drive, Mail Stop 161-213
Pasadena, CA, 91109
+1(818)354-2075 phone
+1(818)393-6875 fax
-----Original Message-----
From: beowulf-***@beowulf.org [mailto:beowulf-***@beowulf.org] On Behalf Of Prentice Bisbal
Sent: Tuesday, September 16, 2008 1:36 PM
Cc: Beowulf
Subject: Re: [Beowulf] MS Cray
Post by Gus Correa
Dear Beowulf and COTS fans
http://www.theregister.co.uk/2008/09/16/cray_baby_super/
IGIDH (I guess it doesn't help.)
Gus Correa
Quote from article:

"It's also attempting to lure scientists and researchers with discretionary IT budgets to forget using shared, giant clusters and get their own box and tuck it in behind their desk where no one can see it to run their workloads locally. The personal supercomputer is not a new idea, but this is the first time that Cray is trying it out in the market."

That will work great until the newbie scientists find that airflow into a computer tucked in "behind their desk where no one can see it" is piss poor, and that fans powerful enough to provide adequate airflow "behind the desk where no one can see it" are going to be LOUD.
-------------------

Well, plus ca change, plus c'est la meme chose..

The same argument was made when the IBM1130 came out, or the IBM 5150, for that matter..

It doesn't have to be loud, and the thermal issue can be dealt with in a variety of ways (some easier and more practical than others).. I could probably support a kW load in my office without causing huge problems (especially if I leave the lights off), and the amount of computation you can get for a kilowatt is always growing.

There is a huge psychological advantage to having the computer physically under your management and control. You don't have folks trying to "optimize the use of a valuable institutional resource" with scheduling, etc. You might be willing to tolerate a factor of 2 hit in performance for the ability to not have to account for anyone else about how much you're using or not using it.

Jim
Tim Cutts
2008-09-17 06:49:12 UTC
Permalink
Post by Lux, James P
There is a huge psychological advantage to having the computer
physically under your management and control. You don't have folks
trying to "optimize the use of a valuable institutional resource"
with scheduling, etc. You might be willing to tolerate a factor of
2 hit in performance for the ability to not have to account for
anyone else about how much you're using or not using it.
And then they all expect the central systems support group to get it
running for them, and to fix it when it breaks, and to generally
maintain it. Suddenly you have dozens of completely different systems
scattered far and wide across your site, and you're starting to get
complaints that the support group are unobtainable these days -
they're never at their desk any more, and don't seem to have any time
to build new stuff any more.

Tim
--
The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
Gus Correa
2008-09-16 22:18:52 UTC
Permalink
Hi Prentice and Beowulf fans
Post by Prentice Bisbal
Post by Gus Correa
Dear Beowulf and COTS fans
http://www.theregister.co.uk/2008/09/16/cray_baby_super/
IGIDH (I guess it doesn't help.)
Gus Correa
"It's also attempting to lure scientists and researchers with
discretionary IT budgets to forget using shared, giant clusters and get
their own box and tuck it in behind their desk where no one can see it
to run their workloads locally. The personal supercomputer is not a new
idea, but this is the first time that Cray is trying it out in the market."
I guess there are and there have been other competitors for this niche
market,
although maybe not with so marketable logos and brand names,
with slightly different scope, etc.
Two recent examples:

http://sicortex.com/products/sc072_pds
http://www.nvidia.com/object/tesla_d870.html

Well, who knows, maybe beowulfs will dwindle,
and products like these will become the HPC mainstream.
Windows has such a foothold in the computer market that this may prove
to be possible.
Would this be the end of civilization as we know it? (Unix, Linux,
COTS, reading this list, ...)
Or would it be replaced by a new state of affairs,
a move towards HPC machines with proprietary design and proprietary
software,
after which we would perhaps be back again to an open architecture?
Anyway, economic cycles may not be this much cyclic.
Post by Prentice Bisbal
That will work great until the newbie scientists find that airflow into
a computer tucked in "behind their desk where no one can see it" is piss
poor, and that fans powerful enough to provide adequate airflow "behind
the desk where no one can see it" are going to be LOUD.
To their credit, they seem to be aware of the noise problem.
Quoting the article:

"Because the CX1 sits in an office environment, the front of the chassis
has an optional noise cancellation add-on, which drops the whirring of
fan noise down to the point where it is actually legal to put it in an
office environment."

Otherwise, your "newbie scientist" can put his/her earbuds and pump up
the volume on his Ipod,
while he/she navigates through the Vista colorful 3D menus.

I still think that there are savings, and perhaps some virtue,
in assembling some components and replacing parts by myself, with a
simple screwdriver.
Or at least to be able to do so, to have this potential.
But maybe this is just wishful romantic thinking.

Gus Correa
--
---------------------------------------------------------------------
Gustavo J. Ponce Correa, PhD - Email: ***@ldeo.columbia.edu
Lamont-Doherty Earth Observatory - Columbia University
P.O. Box 1000 [61 Route 9W] - Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------
Joe Landman
2008-09-16 22:39:20 UTC
Permalink
Post by Gus Correa
Otherwise, your "newbie scientist" can put his/her earbuds and pump up
the volume on his Ipod,
while he/she navigates through the Vista colorful 3D menus.
Owie .... I can just imagine the folks squawking about this at SC08 "Yes
folks, you need a Cray supercomputer to make Vista run at acceptable
performance ..."


The machine seems to run w2k8. My own experience with w2k8 is that,
frankly, it doesn't suck. This is the first time I have seen a windows
release that I can say that about.

The low end economics probably won't work out for this machine though,
unless it is N times faster than some other agglomeration of Intel-like
products. Adding windows will add cost, not performance in any
noticeable way.

The question that Cray (and every other vendor building non-commodity
units) is how much better is this than a small cluster someone can
build/buy on their own? Better as in faster, able to leap more tall
buildings in a single bound, ... (Superman TV show reference for those
not in the know). And the hard part will be justifying the additional
cost. If the machine isn't 2x the performance, would it be able to
justify 2x the price? Since it appears to be a somewhat well branded
cluster, I am not sure that argument will be easy to make.
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: ***@scalableinformatics.com
web : http://www.scalableinformatics.com
http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
Gus Correa
2008-09-17 00:10:59 UTC
Permalink
Hi Joe and fellow Beowulf fans
Post by Joe Landman
Post by Gus Correa
Otherwise, your "newbie scientist" can put his/her earbuds and pump
up the volume on his Ipod,
while he/she navigates through the Vista colorful 3D menus.
Owie .... I can just imagine the folks squawking about this at SC08
"Yes folks, you need a Cray supercomputer to make Vista run at
acceptable performance ..."
:)
Post by Joe Landman
The machine seems to run w2k8. My own experience with w2k8 is that,
frankly, it doesn't suck. This is the first time I have seen a
windows release that I can say that about.
The low end economics probably won't work out for this machine though,
unless it is N times faster than some other agglomeration of
Intel-like products. Adding windows will add cost, not performance in
any noticeable way.
What if performance is not the main goal?
Here is what the article has to say about it:

"Microsoft's strategy - one that no supercomputer maker and no X64 chip
maker can ignore - is to attack from the bottom, to find those myriad
new HPC users who never learned Unix, never learned Linux, and have no
desire to."

There have been several long and heated discussions on this list about
computer literacy and computer education for scientists and science
students.
Mostly centered on computer languages, not so much was said about
Unix/Linux proficiency,
bits of shell or scripting language skills, and the rudiments of
Unix/Linux tools
and programming environment.
I don't intend to reopen them.
However, was Microsoft listening to those discussions?
Post by Joe Landman
The question that Cray (and every other vendor building non-commodity
units) is how much better is this than a small cluster someone can
build/buy on their own? Better as in faster, able to leap more tall
buildings in a single bound, ... (Superman TV show reference for those
not in the know). And the hard part will be justifying the additional
cost. If the machine isn't 2x the performance, would it be able to
justify 2x the price? Since it appears to be a somewhat well branded
cluster, I am not sure that argument will be easy to make.
You are right about the economics, at least if we consider hardware alone.
According to the article the full configuration has 64 Xeon 3.4GHZ cores,
equivalent to eight cluster nodes with IB hardware.
The "fully loaded" machine price is $80k, or $10k per node.

Quoting from the article:

"A single chassis can house a maximum of 4 TB of disk or -when using the
fastest 3.4 GHz quad-core Xeons Intel has delivered - up to 768
gigaflops of computing power in a single chassis. (That's eight
two-socket blades using quad-core Xeons, for a total of 64 cores).
Obviously, three of these CX1s linked up yields 2.3 teraflops - a nice
size for a personal super."

"The base price of the chassis with bare bones blades and switches is
$25,000. When the machine is fully loaded, the price tag comes to around
$80,000 or so. Cray is selling the CX1 boxes online starting today - the
first time a Cray machine has been sold online and directly - and
expects to have volume shipments revved up by the end of October."

***

Here is the link to the CX1 on the Cray web site:

http://www.cray.com/products/CX1.aspx

You need MS Explorer to customize/price it.

Gus Correa
Gerry Creager
2008-09-17 13:22:31 UTC
Permalink
Post by Gus Correa
Hi Joe and fellow Beowulf fans
Post by Joe Landman
Post by Gus Correa
Otherwise, your "newbie scientist" can put his/her earbuds and pump
up the volume on his Ipod,
while he/she navigates through the Vista colorful 3D menus.
Owie .... I can just imagine the folks squawking about this at SC08
"Yes folks, you need a Cray supercomputer to make Vista run at
acceptable performance ..."
:)
Post by Joe Landman
The machine seems to run w2k8. My own experience with w2k8 is that,
frankly, it doesn't suck. This is the first time I have seen a
windows release that I can say that about.
The low end economics probably won't work out for this machine though,
unless it is N times faster than some other agglomeration of
Intel-like products. Adding windows will add cost, not performance in
any noticeable way.
What if performance is not the main goal?
"Microsoft's strategy - one that no supercomputer maker and no X64 chip
maker can ignore - is to attack from the bottom, to find those myriad
new HPC users who never learned Unix, never learned Linux, and have no
desire to."
There have been several long and heated discussions on this list about
computer literacy and computer education for scientists and science
students. Mostly centered on computer languages, not so much was said
about Unix/Linux proficiency,
bits of shell or scripting language skills, and the rudiments of
Unix/Linux tools
and programming environment.
I don't intend to reopen them.
However, was Microsoft listening to those discussions?
Post by Joe Landman
The question that Cray (and every other vendor building non-commodity
units) is how much better is this than a small cluster someone can
build/buy on their own? Better as in faster, able to leap more tall
buildings in a single bound, ... (Superman TV show reference for those
not in the know). And the hard part will be justifying the additional
cost. If the machine isn't 2x the performance, would it be able to
justify 2x the price? Since it appears to be a somewhat well branded
cluster, I am not sure that argument will be easy to make.
You are right about the economics, at least if we consider hardware alone.
According to the article the full configuration has 64 Xeon 3.4GHZ cores,
equivalent to eight cluster nodes with IB hardware.
The "fully loaded" machine price is $80k, or $10k per node.
"A single chassis can house a maximum of 4 TB of disk or -when using the
fastest 3.4 GHz quad-core Xeons Intel has delivered - up to 768
gigaflops of computing power in a single chassis. (That's eight
two-socket blades using quad-core Xeons, for a total of 64 cores).
Obviously, three of these CX1s linked up yields 2.3 teraflops - a nice
size for a personal super."
"The base price of the chassis with bare bones blades and switches is
$25,000. When the machine is fully loaded, the price tag comes to around
$80,000 or so. Cray is selling the CX1 boxes online starting today - the
first time a Cray machine has been sold online and directly - and
expects to have volume shipments revved up by the end of October."
***
http://www.cray.com/products/CX1.aspx
You need MS Explorer to customize/price it.
I just knew you had to be wrong, but sure enough, I can't see config
options. It's a show stopper for me. If I need IE to buy the system,
it's not likely to happen until A) there's an IE that runs natively on
*nix, and B) it doesn't have the myriad problems associated with IE in
the past.

I do admit to a sinking feeling when I noted that the front page (and of
course, the subsequent pages) were ASPX...

I suspect Microsoft has been listening here. I also suspect this
machine will do ok in the business world, but somehow I doubt they're
gonna see significant headway in a lot of the scientific arenas. If you
aren't computer literate, you're not likely to port a complicated model
from *nix to Windows, nor are you likely to write a significant piece of
code. I've a geodesist friend who DOES write solely for Windows, but
that's a conscious choice by someone who was a talented computer
scientist first, and a geodesist later in life. He uses Windows
because, well, mainly because the folks he teaches, and writes code for,
do. However, he's the exception.

The CX1 looks like something I'd love next to my desk -- with Linux on
it -- to accomplish testing before I take something to the big iron. It
might even allow me to pre- and post-process my data for hurricane WRF
runs. It's not hefty enough to let me do those runs in the timeframe I
require otherwise.

It's a tool, not a solution.

gerry
--
Gerry Creager -- ***@tamu.edu
Texas Mesonet -- AATLT, Texas A&M University
Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983
Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843
Joe Landman
2008-09-17 14:01:54 UTC
Permalink
Post by Gerry Creager
The CX1 looks like something I'd love next to my desk -- with Linux on
it -- to accomplish testing before I take something to the big iron. It
This is something I suspect you will be able to do. The CX1 may support
Linux (and it wouldn't surprise me if it had that as an option).
Post by Gerry Creager
might even allow me to pre- and post-process my data for hurricane WRF
runs. It's not hefty enough to let me do those runs in the timeframe I
require otherwise.
Heh... We like the under-desktop experience, with lots of fast disk and
big pipes to the disk. Honestly, this looks like the direction for most
of "smaller" HPC that can run locally under your own control. The big
iron/heavy metal for the large (non-prototype) jobs.
Post by Gerry Creager
It's a tool, not a solution.
Yup. Lots of folks get lost in this, thinking that a solution == the
thing they market. Its not. It is just one aspect of things. A
product is a tool. A solution is so much more than that (and usually
starts with a statement of a problem ... otherwise it is a solution
searching for a problem).

Joe
Post by Gerry Creager
gerry
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: ***@scalableinformatics.com
web : http://www.scalableinformatics.com
http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
John Leidel
2008-09-17 14:21:43 UTC
Permalink
Post by Joe Landman
Post by Gerry Creager
The CX1 looks like something I'd love next to my desk -- with Linux on
it -- to accomplish testing before I take something to the big iron. It
This is something I suspect you will be able to do. The CX1 may support
Linux (and it wouldn't surprise me if it had that as an option).
Indeed... it supports RedHat. Oddly enough, no mention of SLES. Cray
has been running SLES on their XT login/service nodes for quite some
time. I'm curious why they changed horses.
Post by Joe Landman
Post by Gerry Creager
might even allow me to pre- and post-process my data for hurricane WRF
runs. It's not hefty enough to let me do those runs in the timeframe I
require otherwise.
Heh... We like the under-desktop experience, with lots of fast disk and
big pipes to the disk. Honestly, this looks like the direction for most
of "smaller" HPC that can run locally under your own control. The big
iron/heavy metal for the large (non-prototype) jobs.
Post by Gerry Creager
It's a tool, not a solution.
Yup. Lots of folks get lost in this, thinking that a solution == the
thing they market. Its not. It is just one aspect of things. A
product is a tool. A solution is so much more than that (and usually
starts with a statement of a problem ... otherwise it is a solution
searching for a problem).
Joe
Post by Gerry Creager
gerry
Gus Correa
2008-09-17 16:32:33 UTC
Permalink
Dear Beowulf fans

Since I posted the Cray CX1 announcement,
just to be fair to other players, here are some of them:

1) SiCortex has a Linux and MIPS (72 processors)
based "deskside supercomputer".
They claim it to work with 300W of power.
Of course, being Linux, it requires Linux literacy to use.
See:

http://sicortex.com/products/sc072_pds/sc072_pds_datasheet

2) NVidia is advertising its Tesla series,
although this GPU-based deskside system will probably work
as a "deskside co-processor" rather than as a "deskside supercomputer".
Literacy in CUDA, not only in C and Linux, is probably required to use
it effectively.
GPU experts, please correct me if I am wrong.
See:

http://www.nvidia.com/object/tesla_8_series.html

There may be more "deskside supercomputers" out there,
and I apologize to anyone that may have been omitted.
NEC had something called SX-8i, I think, not very long ago.
If the Cray CX1 idea sticks and the machines sell, it is likely that other
companies will launch similar product lines.

***

Deskside supercomputers, and even bigger ones,
are starting to be marketed as "plug-and-play", as something that
requires little, if any,
system administration and maintenance
(proprietary hardware and maintenance fees are rarely mentioned),
and not much computer literacy to be used.
Other postings to this thread already pointed this out.
All they need is an available power outlet on your office wall ( ... how
about an Ethernet port?),
and similar marketing arguments.

They are marketed in contrast to clusters,
which are pictured as complicated beasts, hard and expensive to maintain,
requiring dedicated IT personnel, sucking more power, and leading to
higher TCO.
The logic presented to decision makers would be that, besides being user
friendly,
what you pay upfront for these machines you recover quickly in IT
salaries and utility bills.

Gus Correa
Post by John Leidel
Post by Joe Landman
Post by Gerry Creager
The CX1 looks like something I'd love next to my desk -- with Linux on
it -- to accomplish testing before I take something to the big iron. It
This is something I suspect you will be able to do. The CX1 may support
Linux (and it wouldn't surprise me if it had that as an option).
Indeed... it supports RedHat. Oddly enough, no mention of SLES. Cray
has been running SLES on their XT login/service nodes for quite some
time. I'm curious why they changed horses.
Post by Joe Landman
Post by Gerry Creager
might even allow me to pre- and post-process my data for hurricane WRF
runs. It's not hefty enough to let me do those runs in the timeframe I
require otherwise.
Heh... We like the under-desktop experience, with lots of fast disk and
big pipes to the disk. Honestly, this looks like the direction for most
of "smaller" HPC that can run locally under your own control. The big
iron/heavy metal for the large (non-prototype) jobs.
Post by Gerry Creager
It's a tool, not a solution.
Yup. Lots of folks get lost in this, thinking that a solution == the
thing they market. Its not. It is just one aspect of things. A
product is a tool. A solution is so much more than that (and usually
starts with a statement of a problem ... otherwise it is a solution
searching for a problem).
Joe
Post by Gerry Creager
gerry
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
Joe Landman
2008-09-17 17:06:19 UTC
Permalink
Post by Gus Correa
Dear Beowulf fans
Since I posted the Cray CX1 announcement,
1) SiCortex has a Linux and MIPS (72 processors)
based "deskside supercomputer".
They claim it to work with 300W of power.
Of course, being Linux, it requires Linux literacy to use.
http://sicortex.com/products/sc072_pds/sc072_pds_datasheet
Contrary to various public statements from some vendors, we have seen
linux literacy on the rise, and have seen a number of organizations
switch all their staff to it. It turns out for them to be easier/lower
cost to deploy and support. I am not at liberty to say who though.
Post by Gus Correa
2) NVidia is advertising its Tesla series,
although this GPU-based deskside system will probably work
as a "deskside co-processor" rather than as a "deskside supercomputer".
Literacy in CUDA, not only in C and Linux, is probably required to use
it effectively.
GPU experts, please correct me if I am wrong.
http://www.nvidia.com/object/tesla_8_series.html
CUDA is needed to program it. You can use CUDA enabled software without
knowing how to program in CUDA.
Post by Gus Correa
There may be more "deskside supercomputers" out there,
and I apologize to anyone that may have been omitted.
NEC had something called SX-8i, I think, not very long ago.
If the Cray CX1 idea sticks and the machines sell, it is likely that other
companies will launch similar product lines.
The personal super has been around a while as an idea. Some versions of
it sorta kinda work, but not as a market/product.
Post by Gus Correa
***
Deskside supercomputers, and even bigger ones,
are starting to be marketed as "plug-and-play", as something that
requires little, if any,
system administration and maintenance
(proprietary hardware and maintenance fees are rarely mentioned),
and not much computer literacy to be used.
The appliance model. Great if you can get it to work in the market, but
many codes are quite different, so you get the choice between a 1-off
accelerator (won't sell), or the very general purpose black box (which
often won't run the ISV codes without modification of the basic black box).
Post by Gus Correa
Other postings to this thread already pointed this out.
All they need is an available power outlet on your office wall ( ... how
about an Ethernet port?),
and similar marketing arguments.
They are marketed in contrast to clusters,
which are pictured as complicated beasts, hard and expensive to maintain,
requiring dedicated IT personnel, sucking more power, and leading to
higher TCO.
They really aren't as terrible as this. We (and a number of others)
make clusters that are pretty much plug-n-play. You configure/order it,
and it gets delivered, and works after assembly (connection to wall,
network, cooling). Some require more remote assembly than others, some
you can just ship a bunch-o-boxes.
Post by Gus Correa
The logic presented to decision makers would be that, besides being user
friendly,
what you pay upfront for these machines you recover quickly in IT
salaries and utility bills.
Hmmm.... some of these arguments are valid, some are not.
Post by Gus Correa
Gus Correa
Post by John Leidel
Post by Joe Landman
Post by Gerry Creager
The CX1 looks like something I'd love next to my desk -- with Linux
on it -- to accomplish testing before I take something to the big
iron. It
This is something I suspect you will be able to do. The CX1 may
support Linux (and it wouldn't surprise me if it had that as an option).
Indeed... it supports RedHat. Oddly enough, no mention of SLES. Cray
has been running SLES on their XT login/service nodes for quite some
time. I'm curious why they changed horses.
Post by Joe Landman
Post by Gerry Creager
might even allow me to pre- and post-process my data for hurricane
WRF runs. It's not hefty enough to let me do those runs in the
timeframe I require otherwise.
Heh... We like the under-desktop experience, with lots of fast disk
and big pipes to the disk. Honestly, this looks like the direction
for most of "smaller" HPC that can run locally under your own
control. The big iron/heavy metal for the large (non-prototype) jobs.
Post by Gerry Creager
It's a tool, not a solution.
Yup. Lots of folks get lost in this, thinking that a solution == the
thing they market. Its not. It is just one aspect of things. A
product is a tool. A solution is so much more than that (and usually
starts with a statement of a problem ... otherwise it is a solution
searching for a problem).
Joe
Post by Gerry Creager
gerry
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: ***@scalableinformatics.com
web : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax : +1 734 786 8452
cell : +1 734 612 4615
Joshua Baker-LePain
2008-09-17 16:24:16 UTC
Permalink
Post by Gus Correa
http://www.cray.com/products/CX1.aspx
You need MS Explorer to customize/price it.
I just knew you had to be wrong, but sure enough, I can't see config options.
It's a show stopper for me. If I need IE to buy the system, it's not likely
to happen until A) there's an IE that runs natively on *nix, and B) it
doesn't have the myriad problems associated with IE in the past.
Hrm -- WORKSFORME using FF3 on Fedora 9. Notably, Cray is happy to sell
the CX1 to you sans OS. W2K8 is a $3752 option (at least in my sample
config with 8 compute blades). As was noted, the Linux they suggest is
"Red Hat Linux", by which one would assume they mean RHEL (why does
everybody make that mistake -- there hasn't been a "Red Hat Linux" since
RH9 (shudder)).

Also, as one would expect, the hardware premium is hefty. A compute blade
with dual Xeon E5462s, 16GB RAM (8x2GB), and an 80GB HDD is $6656.
Without even trying too hard I can get a similarly configured 1U node for
$4400. So that's a 50% markup on nodes, not to mention the almost $9K for
the chassis. Of course, that's before any discounts.

Maybe it's my miserly ways, but I just don't see the value proposition
here...
--
Joshua Baker-LePain
QB3 Shared Cluster Sysadmin
UCSF
Joe Landman
2008-09-17 16:58:27 UTC
Permalink
Post by Joshua Baker-LePain
Also, as one would expect, the hardware premium is hefty. A compute
blade with dual Xeon E5462s, 16GB RAM (8x2GB), and an 80GB HDD is $6656.
Without even trying too hard I can get a similarly configured 1U node
for $4400. So that's a 50% markup on nodes, not to mention the almost
$9K for the chassis. Of course, that's before any discounts.
Maybe it's my miserly ways, but I just don't see the value proposition
here...
What about a bundled bumper sticker saying something like "my other
computer is a Cray" ... Thats gotta be worth something... right?

Way way back I remember internal discussions in SGI (or SGI/Cray if you
prefer) that went something like "you can't paint a box purple and
charge 3x the price". Some thought you could.
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: ***@scalableinformatics.com
web : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax : +1 734 786 8452
cell : +1 734 612 4615
Lux, James P
2008-09-17 17:20:30 UTC
Permalink
From: beowulf-***@beowulf.org [mailto:beowulf-***@beowulf.org] On Behalf Of Joshua Baker-LePain
Sent: Wednesday, September 17, 2008 9:24 AM
To: Gerry Creager
Cc: Beowulf
Subject: Re: [Beowulf] MS Cray
Post by Gus Correa
http://www.cray.com/products/CX1.aspx
You need MS Explorer to customize/price it.
I just knew you had to be wrong, but sure enough, I can't see config options.
It's a show stopper for me. If I need IE to buy the system, it's not
likely to happen until A) there's an IE that runs natively on *nix,
and B) it doesn't have the myriad problems associated with IE in the past.
Hrm -- WORKSFORME using FF3 on Fedora 9. Notably, Cray is happy to sell the CX1 to you sans OS. W2K8 is a $3752 option (at least in my sample config with 8 compute blades). As was noted, the Linux they suggest is "Red Hat Linux", by which one would assume they mean RHEL (why does everybody make that mistake -- there hasn't been a "Red Hat Linux" since
RH9 (shudder)).

Also, as one would expect, the hardware premium is hefty. A compute blade with dual Xeon E5462s, 16GB RAM (8x2GB), and an 80GB HDD is $6656.
Without even trying too hard I can get a similarly configured 1U node for $4400. So that's a 50% markup on nodes, not to mention the almost $9K for the chassis. Of course, that's before any discounts.

Maybe it's my miserly ways, but I just don't see the value proposition here...
--

Noise?
Packaging?
Nobody ever got fired for buying Cray?
Gus Correa
2008-09-17 18:14:46 UTC
Permalink
Hi Gerry and Beowulf fans
Post by Gerry Creager
Post by Gus Correa
http://www.cray.com/products/CX1.aspx
You need MS Explorer to customize/price it.
I just knew you had to be wrong, but sure enough, I can't see config
options.
Thanks for your trust!
Post by Gerry Creager
It's a show stopper for me. If I need IE to buy the system, it's not
likely to happen until A) there's an IE that runs natively on *nix,
and B) it doesn't have the myriad problems associated with IE in the
past.
BTW, the Cray web site was changed today,
and now I can configure/price the CX1 from Linux/Firefox.

Gus Correa
Post by Gerry Creager
I do admit to a sinking feeling when I noted that the front page (and
of course, the subsequent pages) were ASPX...
I suspect Microsoft has been listening here. I also suspect this
machine will do ok in the business world, but somehow I doubt they're
gonna see significant headway in a lot of the scientific arenas. If
you aren't computer literate, you're not likely to port a complicated
model from *nix to Windows, nor are you likely to write a significant
piece of code. I've a geodesist friend who DOES write solely for
Windows, but that's a conscious choice by someone who was a talented
computer scientist first, and a geodesist later in life. He uses
Windows because, well, mainly because the folks he teaches, and writes
code for, do. However, he's the exception.
The CX1 looks like something I'd love next to my desk -- with Linux on
it -- to accomplish testing before I take something to the big iron.
It might even allow me to pre- and post-process my data for hurricane
WRF runs. It's not hefty enough to let me do those runs in the
timeframe I require otherwise.
It's a tool, not a solution.
gerry
Eric Thibodeau
2008-09-17 19:18:06 UTC
Permalink
Post by Gus Correa
Hi Gerry and Beowulf fans
Post by Gerry Creager
Post by Gus Correa
http://www.cray.com/products/CX1.aspx
You need MS Explorer to customize/price it.
I just knew you had to be wrong, but sure enough, I can't see config
options.
Thanks for your trust!
Post by Gerry Creager
It's a show stopper for me. If I need IE to buy the system, it's not
likely to happen until A) there's an IE that runs natively on *nix,
and B) it doesn't have the myriad problems associated with IE in the
past.
BTW, the Cray web site was changed today,
and now I can configure/price the CX1 from Linux/Firefox.
I think I heard the "Oh crap!" from Cray from here when one of their
employees must have noticed the remarks on the BW ml ;)...this might
also explain why I was hitting errors on the page these past 2
days...nice QA ;)
Post by Gus Correa
Gus Correa
Post by Gerry Creager
I do admit to a sinking feeling when I noted that the front page (and
of course, the subsequent pages) were ASPX...
I suspect Microsoft has been listening here. I also suspect this
machine will do ok in the business world, but somehow I doubt they're
gonna see significant headway in a lot of the scientific arenas. If
you aren't computer literate, you're not likely to port a complicated
model from *nix to Windows, nor are you likely to write a significant
piece of code. I've a geodesist friend who DOES write solely for
Windows, but that's a conscious choice by someone who was a talented
computer scientist first, and a geodesist later in life. He uses
Windows because, well, mainly because the folks he teaches, and
writes code for, do. However, he's the exception.
The CX1 looks like something I'd love next to my desk -- with Linux
on it -- to accomplish testing before I take something to the big
iron. It might even allow me to pre- and post-process my data for
hurricane WRF runs. It's not hefty enough to let me do those runs in
the timeframe I require otherwise.
It's a tool, not a solution.
gerry
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
Lux, James P
2008-09-17 20:14:21 UTC
Permalink
Post by Gerry Creager
Post by Gerry Creager
I suspect Microsoft has been listening here. I also suspect this
machine will do ok in the business world, but somehow I
doubt they're
Post by Gerry Creager
gonna see significant headway in a lot of the scientific
arenas.
Of course MS is on the list. Why not? Look back through the archives when CCS was being discussed. And if MS wants to develop products to address some specific market niche, more power to them.
Eric Thibodeau
2008-09-17 20:27:54 UTC
Permalink
Post by Lux, James P
Post by Gerry Creager
Post by Gerry Creager
I suspect Microsoft has been listening here. I also suspect this
machine will do ok in the business world, but somehow I
doubt they're
Post by Gerry Creager
gonna see significant headway in a lot of the scientific
arenas.
Of course MS is on the list. Why not? Look back through the archives when CCS was being discussed. And if MS wants to develop products to address some specific market niche, more power to them.
Not that this was one of my comments, the MS dude never hid, he actually
posted explicit questions regarding how MS should approach the
clustering community ;)

On that node, here are my questions to the MS implementation of
clustering (and how CX1 will actually be usable or how one will be able
to develop _for_ execution on one of the MS based CX1):

Currently, most of our users are under Linux on their WS so that they
can develop something that will potentially run off a big cluster. This
implies the users are under a POSIX compliant OS with good
MPI/OpenMP/threading support, which waters down to Linux or OSX (those
Power Macs are impressive and *silent*).

Where I have reserves about the MS solution is as follows: how will MS
users be able to develop parallel code locally on their WS without
needing to upgrade/change their hardware/OS to be compatible with a MS
based cluster/CX1. It's not made clear weather the "clustering tools"
are tightly integrated into the CX1 platform or if, as with Linux or
OSX, it's a simple case of installing mpi-ish libs (and a few others).

With the pricing scheme, I can't imagine _every_ dev getting their own
CX1 to play on so I believe adoption of the platform required ease of
installation (ie: all tools should be available throughout windows
2000-xp-vista) and shouldn't even be version specific (IMHO)...but
that's me being used to Linux heh! ;)

Eric
Eric Thibodeau
2008-09-18 02:57:36 UTC
Permalink
Eric,
I don’t have a CX1 (yet!) but I expect the development experience will
be the usual Windows HPC model. Design/code/test parallel code in
Visual Studio then do test runs on the cluster. With Visual C++ and
Windows HPC, you have MPI, OpenMP, threading, and an MPI debugger. You
can debug and test your code either locally or on a cluster from the
same development environment.
http://www.pluralsight.com/community/blogs/drjoe/archive/2008/06/19/51178.aspx
It is definitely our goal to make it simple to develop an MPI
application on your desktop and then easily run it, tune it, debug it
on a cluster of any size.
John Vert
Windows HPC
Yes but...that brings out my point, this is implying that _everyone_
would need HPC server. And seeing how much that costs for the CX1, I'm
curious to see how the research departments will be able to cope with
the licensing issues and TCO of using the MS solution compared to the
FOSS and/or POSIX proprietary approach. There is also a question of
readily available resources that will be able to run the code MS
flavored code, there are many MPI+posix clusters out there, none that I
know of that run MS HPC.
*Sent:* Wednesday, September 17, 2008 1:28 PM
*To:* Lux, James P
*Cc:* Beowulf
*Subject:* Re: [Beowulf] MS Cray
On that node, here are my questions to the MS implementation of
clustering (and how CX1 will actually be usable or how one will be
Currently, most of our users are under Linux on their WS so that they
can develop something that will potentially run off a big cluster.
This implies the users are under a POSIX compliant OS with good
MPI/OpenMP/threading support, which waters down to Linux or OSX (those
Power Macs are impressive and *silent*).
Where I have reserves about the MS solution is as follows: how will MS
users be able to develop parallel code locally on their WS without
needing to upgrade/change their hardware/OS to be compatible with a MS
based cluster/CX1. It's not made clear weather the "clustering tools"
are tightly integrated into the CX1 platform or if, as with Linux or
OSX, it's a simple case of installing mpi-ish libs (and a few others).
With the pricing scheme, I can't imagine _every_ dev getting their own
CX1 to play on so I believe adoption of the platform required ease of
installation (ie: all tools should be available throughout windows
2000-xp-vista) and shouldn't even be version specific (IMHO)...but
that's me being used to Linux heh! ;)
Eric
Shahrokh Mortazavi
2008-09-18 16:26:13 UTC
Permalink
I will add to what john said that we’re working w leading Linux/Unix tool vendors to port their offerings to WHPC. For example key math libs are already available, tools like Vampir are about to go Beta (uses the moral equivalent of Dtrace on Windows for tracing), Allinea has created a Visual Studio add-in for debugging on a cluster, etc. WHPC also has an “SOA” programming model that’s useful for certain verticals (such as finserv). You should also check out MPI.Net, it’s a really well done high performance wrapper around MPI (esp if you were put off by previous .Net/Java + MPI attempts) – try using it from C#/F#/Python. On the Fortran side, both PGI and Intel make great compilers, both of which are available as snap-ins for Visual Studio w full project/debugger/profiler/etc support.

S


From: beowulf-***@beowulf.org [mailto:beowulf-***@beowulf.org] On Behalf Of Eric Thibodeau
Sent: Wednesday, September 17, 2008 7:58 PM
To: John Vert
Cc: Beowulf
Subject: Re: [Beowulf] MS Cray

John Vert wrote:
Eric,

I don’t have a CX1 (yet!) but I expect the development experience will be the usual Windows HPC model. Design/code/test parallel code in Visual Studio then do test runs on the cluster. With Visual C++ and Windows HPC, you have MPI, OpenMP, threading, and an MPI debugger. You can debug and test your code either locally or on a cluster from the same development environment.

If you are interested, Pluralsight has a very detailed tutorial here: http://www.pluralsight.com/community/blogs/drjoe/archive/2008/06/19/51178.aspx

It is definitely our goal to make it simple to develop an MPI application on your desktop and then easily run it, tune it, debug it on a cluster of any size.

John Vert
Windows HPC
Yes but...that brings out my point, this is implying that _everyone_ would need HPC server. And seeing how much that costs for the CX1, I'm curious to see how the research departments will be able to cope with the licensing issues and TCO of using the MS solution compared to the FOSS and/or POSIX proprietary approach. There is also a question of readily available resources that will be able to run the code MS flavored code, there are many MPI+posix clusters out there, none that I know of that run MS HPC.


From: beowulf-***@beowulf.org [mailto:beowulf-***@beowulf.org] On Behalf Of Eric Thibodeau
Sent: Wednesday, September 17, 2008 1:28 PM
To: Lux, James P
Cc: Beowulf
Subject: Re: [Beowulf] MS Cray

On that node, here are my questions to the MS implementation of clustering (and how CX1 will actually be usable or how one will be able to develop _for_ execution on one of the MS based CX1):

Currently, most of our users are under Linux on their WS so that they can develop something that will potentially run off a big cluster. This implies the users are under a POSIX compliant OS with good MPI/OpenMP/threading support, which waters down to Linux or OSX (those Power Macs are impressive and *silent*).

Where I have reserves about the MS solution is as follows: how will MS users be able to develop parallel code locally on their WS without needing to upgrade/change their hardware/OS to be compatible with a MS based cluster/CX1. It's not made clear weather the "clustering tools" are tightly integrated into the CX1 platform or if, as with Linux or OSX, it's a simple case of installing mpi-ish libs (and a few others).

With the pricing scheme, I can't imagine _every_ dev getting their own CX1 to play on so I believe adoption of the platform required ease of installation (ie: all tools should be available throughout windows 2000-xp-vista) and shouldn't even be version specific (IMHO)...but that's me being used to Linux heh! ;)

Eric
Steffen Persvold
2008-09-17 19:39:34 UTC
Permalink
Post by Eric Thibodeau
Post by Gus Correa
BTW, the Cray web site was changed today,
and now I can configure/price the CX1 from Linux/Firefox.
I think I heard the "Oh crap!" from Cray from here when one of their
employees must have noticed the remarks on the BW ml ;)...this might
also explain why I was hitting errors on the page these past 2
days...nice QA ;)
The Cray website is currently down for all practical purposes. Even with
IE7 the site just rarely loads correctly; 90% of the time one of the
pane buttons on the site (Products, Support etc.) is missing the bitmap
file etc... Other times you just get a "HTTP Error 503 - Service
unavailable" error. I guess that's what you get with a M$ web server, ey ?

cheers,
--SP
Matt Lawrence
2008-09-18 04:24:56 UTC
Permalink
The CX1 looks like something I'd love next to my desk -- with Linux on it --
to accomplish testing before I take something to the big iron. It might even
allow me to pre- and post-process my data for hurricane WRF runs. It's not
hefty enough to let me do those runs in the timeframe I require otherwise.
Please don't! If you tuned that thing on, it would probably blow the
breaker for my office as well.

I'll bet we could put something together that is faster for less than half
the price.

-- Matt
It's not what I know that counts.
It's what I can remember in time to use.
Greg Lindahl
2008-09-17 01:40:15 UTC
Permalink
Post by Joe Landman
The question that Cray (and every other vendor building non-commodity
units) is how much better is this than a small cluster someone can
build/buy on their own?
My impression has been that so far, there hasn't been a huge market
discovered for supercomputing appliances. Which is a shame, but there
you have it. It has been tried. And if anyone does make it work, I'm
sure everyone else will be all over it.

Cray also has a well-known inability to sell low-priced systems,
including the Y-MP/EL and XD1. Perhaps the 3rd time's the charm.

-- greg
Eric Thibodeau
2008-09-17 14:13:25 UTC
Permalink
Post by Joe Landman
Post by Gus Correa
Otherwise, your "newbie scientist" can put his/her earbuds and pump
up the volume on his Ipod,
while he/she navigates through the Vista colorful 3D menus.
Owie .... I can just imagine the folks squawking about this at SC08
"Yes folks, you need a Cray supercomputer to make Vista run at
acceptable performance ..."
Maybe they have a "tune options for performance" option ;)
Post by Joe Landman
The machine seems to run w2k8. My own experience with w2k8 is that,
frankly, it doesn't suck. This is the first time I have seen a
windows release that I can say that about.
A few questions (not necessarily expecting a response):

POSIX?
VERBS?
Kernel latency and scheduler control?

These are the real barriers IMHO, without minimally supporting POSIX
(threads), there is very little incentive to use the machine for
development unless you're willing to accept the code will _only_ run on
your "desktop".
Post by Joe Landman
The low end economics probably won't work out for this machine though,
unless it is N times faster than some other agglomeration of
Intel-like products. Adding windows will add cost, not performance in
any noticeable way.
The question that Cray (and every other vendor building non-commodity
units) is how much better is this than a small cluster someone can
build/buy on their own? Better as in faster, able to leap more tall
buildings in a single bound, ... (Superman TV show reference for those
not in the know). And the hard part will be justifying the additional
cost. If the machine isn't 2x the performance, would it be able to
justify 2x the price? Since it appears to be a somewhat well branded
cluster, I am not sure that argument will be easy to make.
I just rebuilt a 32 core cluster for ~5k$ (CAD) (8*Q6600 1Gig RAM/node +
gige netwroking). Bang for the buck? I can't wait to see the CX1's
performance specs under _both_ windows and Linux.

Eric
Joe Landman
2008-09-17 15:54:35 UTC
Permalink
Post by Eric Thibodeau
Post by Joe Landman
Post by Gus Correa
Otherwise, your "newbie scientist" can put his/her earbuds and pump
up the volume on his Ipod,
while he/she navigates through the Vista colorful 3D menus.
Owie .... I can just imagine the folks squawking about this at SC08
"Yes folks, you need a Cray supercomputer to make Vista run at
acceptable performance ..."
Maybe they have a "tune options for performance" option ;)
Post by Joe Landman
The machine seems to run w2k8. My own experience with w2k8 is that,
frankly, it doesn't suck. This is the first time I have seen a
windows release that I can say that about.
POSIX?
VERBS?
Kernel latency and scheduler control?
Don't mistake me for a w2k8 apologist. I reamed them pretty hard on the
lack of a real posix infrastructure (they claim SUA, but frankly it
doesn't build most of what we throw at it, so it really is a non-starter
and not worth considering IMO). They need to pull Cygwin in close and
tight to get a good POSIX infrastructure. It is in their best
interests. Sadly, I suspect the ego driven nature of this will pretty
much prevent them from doing this. Can't touch the "toxic" OSS now, can
they ...

IB Verbs? Well through OFED, yes. Through the windows stack? Who
knows. We were playing with it on JackRabbit for a customer test/benchmark.

Kernel latency? Much better/more responsive than w2k3. Scheduler
control? Not sure how much you have. I don't like deep diving into
registries ... that is a pretty sure way to kill a windows machine.
Post by Eric Thibodeau
These are the real barriers IMHO, without minimally supporting POSIX
(threads), there is very little incentive to use the machine for
development unless you're willing to accept the code will _only_ run on
your "desktop".
Post by Joe Landman
The low end economics probably won't work out for this machine though,
unless it is N times faster than some other agglomeration of
Intel-like products. Adding windows will add cost, not performance in
any noticeable way.
The question that Cray (and every other vendor building non-commodity
units) is how much better is this than a small cluster someone can
build/buy on their own? Better as in faster, able to leap more tall
buildings in a single bound, ... (Superman TV show reference for those
not in the know). And the hard part will be justifying the additional
cost. If the machine isn't 2x the performance, would it be able to
justify 2x the price? Since it appears to be a somewhat well branded
cluster, I am not sure that argument will be easy to make.
I just rebuilt a 32 core cluster for ~5k$ (CAD) (8*Q6600 1Gig RAM/node +
gige netwroking). Bang for the buck? I can't wait to see the CX1's
performance specs under _both_ windows and Linux.
The desktop CPUs/MBs will get you best bang per buck, as long as you
don't mind no ECC, and 8GB ram limits per node. For your applications,
this might be fine. For others, with large memory footprint and long
run times, I see people need/require ECC (as memory density increases,
ECC becomes important .... darned cosmic rays/natural decays/noisy power
supplies/...)
Post by Eric Thibodeau
Eric
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: ***@scalableinformatics.com
web : http://www.scalableinformatics.com
http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
Eric Thibodeau
2008-09-17 17:21:24 UTC
Permalink
Post by Joe Landman
Post by Eric Thibodeau
Post by Joe Landman
Post by Gus Correa
Otherwise, your "newbie scientist" can put his/her earbuds and pump
up the volume on his Ipod,
while he/she navigates through the Vista colorful 3D menus.
Owie .... I can just imagine the folks squawking about this at SC08
"Yes folks, you need a Cray supercomputer to make Vista run at
acceptable performance ..."
Maybe they have a "tune options for performance" option ;)
Post by Joe Landman
The machine seems to run w2k8. My own experience with w2k8 is that,
frankly, it doesn't suck. This is the first time I have seen a
windows release that I can say that about.
POSIX?
VERBS?
Kernel latency and scheduler control?
Don't mistake me for a w2k8 apologist. I reamed them pretty hard on
the lack of a real posix infrastructure (they claim SUA, but frankly
it doesn't build most of what we throw at it, so it really is a
non-starter and not worth considering IMO). They need to pull Cygwin
in close and tight to get a good POSIX infrastructure. It is in their
best interests. Sadly, I suspect the ego driven nature of this will
pretty much prevent them from doing this. Can't touch the "toxic" OSS
now, can they ...
Cygwin...yerk, that emulation bloat is slow as hell and can barely run
some of my basic scripts. A simple find would put the CPU in 100% usage
state. Now I don't blame Cygwin for this per say but rather the way that
windows (probably) runs this as a DOS(ish) app in a somewhat polled
mode. My lack of interest in that OS stopped me from diggin deeper into
why Cygwin is so slow. Given it's a more than mature project, I'd have
expected such poor performance to have been addressed by now.
Post by Joe Landman
IB Verbs? Well through OFED, yes. Through the windows stack? Who
knows. We were playing with it on JackRabbit for a customer
test/benchmark.
...and...the results? ;)
Post by Joe Landman
Kernel latency? Much better/more responsive than w2k3. Scheduler
control? Not sure how much you have. I don't like deep diving into
registries ... that is a pretty sure way to kill a windows machine.
Well, let's just say these are mechanisms I expect an HPC machine to
have when "squeezing the last drop of performance" is mentioned.
Post by Joe Landman
Post by Eric Thibodeau
These are the real barriers IMHO, without minimally supporting POSIX
(threads), there is very little incentive to use the machine for
development unless you're willing to accept the code will _only_ run
on your "desktop".
Post by Joe Landman
The low end economics probably won't work out for this machine
though, unless it is N times faster than some other agglomeration of
Intel-like products. Adding windows will add cost, not performance
in any noticeable way.
The question that Cray (and every other vendor building
non-commodity units) is how much better is this than a small cluster
someone can build/buy on their own? Better as in faster, able to
leap more tall buildings in a single bound, ... (Superman TV show
reference for those not in the know). And the hard part will be
justifying the additional cost. If the machine isn't 2x the
performance, would it be able to justify 2x the price? Since it
appears to be a somewhat well branded cluster, I am not sure that
argument will be easy to make.
I just rebuilt a 32 core cluster for ~5k$ (CAD) (8*Q6600 1Gig
RAM/node + gige netwroking). Bang for the buck? I can't wait to see
the CX1's performance specs under _both_ windows and Linux.
The desktop CPUs/MBs will get you best bang per buck, as long as you
don't mind no ECC, and 8GB ram limits per node. For your
applications, this might be fine. For others, with large memory
footprint and long run times, I see people need/require ECC (as memory
density increases, ECC becomes important .... darned cosmic
rays/natural decays/noisy power supplies/...)
Well, the nodes I built have MBs that state that they can go as high as
8Gigs, but any one with a little experience with the 8Gig mix + 800+MHz
RAM know that very little hardware (MB) can actually do it in a stable
fashion. My recent experiences with "fast" RAM (800 -1066MHz) is that
they end up costing you time ($$$) since it would seem most MBs claim to
support it but they all seem to have some impedance problems of some
sort (totally unstable). And if one reads the fine prints and the QVLs
(Qualifies Vendor Lists), you notice these really high throughputs are
for low memory density of the banks (a _total_ of 2-4 Gigs max). I'd
personally say this is more of an issue than the ECCism of RAM.

I work on "Clusering Algorithms", and not to confuse people, I mean the
k-means type which we could call "data aggregation/mining" algorithms.
They are long running and applied to sizable databases (1.2Gigs) which
need to be loaded onto each node. This is where having multi-core nodes
comes in quite handy as there is way less time lost in data propagation
and loading (the databases).

...which brings me to wonder how the I/O is managed under the CX1...is
it as basic as one I/O node and GFS or do all nodes have their own I/O
paths. I mention this since I've too often seen people ignore the I/O
(load times) ignored in their performance assessments ;)
Andrew M.A. Cater
2008-09-18 06:23:35 UTC
Permalink
Post by Joe Landman
Post by Eric Thibodeau
POSIX?
VERBS?
Kernel latency and scheduler control?
Don't mistake me for a w2k8 apologist. I reamed them pretty hard on the
lack of a real posix infrastructure (they claim SUA, but frankly it
doesn't build most of what we throw at it, so it really is a non-starter
and not worth considering IMO). They need to pull Cygwin in close and
tight to get a good POSIX infrastructure. It is in their best
interests. Sadly, I suspect the ego driven nature of this will pretty
much prevent them from doing this. Can't touch the "toxic" OSS now, can
they ...
It helps when you have a 14 year Linux memory :) A very small company in
UK called Lasermoon run, seemingly, by about 2 people - selling CDs of
Slackware and Linux-FT (which was amazing but too far ahead of its
time).

Ian Nandhra - got the then Linux kernel POSIX certified - which is why
you still see that sometimes on boot-up - ripped the heart out of it and
built utilities with it for Windows. Called it OpenNT (for about a
month, until lawyers got involved.) Changed name to Interix - which was
then bought by Microsoft a few years later and became SFU. The POSIX
layer was grafted into Windows NT. Ian, I think, is somewhere in the US
at a skunkworks but seems to have kept a very low profile/dropped off
the 'Net.
Post by Joe Landman
Post by Eric Thibodeau
I just rebuilt a 32 core cluster for ~5k$ (CAD) (8*Q6600 1Gig RAM/node
+ gige netwroking). Bang for the buck? I can't wait to see the CX1's
performance specs under _both_ windows and Linux.
The desktop CPUs/MBs will get you best bang per buck, as long as you
don't mind no ECC, and 8GB ram limits per node. For your applications,
this might be fine. For others, with large memory footprint and long
run times, I see people need/require ECC (as memory density increases,
ECC becomes important .... darned cosmic rays/natural decays/noisy power
supplies/...)
Cost of a decent motherboard these days with dual core/ dual processors?
-
I may have to replace a very early Tyan with dual Opterons at 1.6G but
have case/disks/reasonable PSU. Damn thing keeps throwing kernel panics and
being temperature sensitive - but it's been my literal under office desk
"supercomputer" for about three years and, because it has wheels,
follows me from desk to desk.

Deskside socket is 230V at 5A and the office has standard aircon: 12 x
Intel 330 dual core Atoms in a lunchbox, anyone?

Andy
Robert G. Brown
2008-09-18 17:54:00 UTC
Permalink
Post by Andrew M.A. Cater
Cost of a decent motherboard these days with dual core/ dual processors?
www.pricewatch.com is one of your many friends here.
Post by Andrew M.A. Cater
I may have to replace a very early Tyan with dual Opterons at 1.6G but
have case/disks/reasonable PSU. Damn thing keeps throwing kernel panics and
being temperature sensitive - but it's been my literal under office desk
"supercomputer" for about three years and, because it has wheels,
follows me from desk to desk.
Nowadays I think a single processor, quad core is likely to be cheapest
per nominal aggregate cycle, although there is also a quad processor,
dual core by somebody -- Supermicro? that looks interesting and might
be a hair cheaper. But of course what processor/core combination is
best for you likely depends on how solidly you're CPU vs memory bound.
Post by Andrew M.A. Cater
Deskside socket is 230V at 5A and the office has standard aircon: 12 x
Intel 330 dual core Atoms in a lunchbox, anyone?
Hmmm, not a lot of headroom there. Better not get a CX1, even as a
gift...;-)

rgb
Post by Andrew M.A. Cater
Andy
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
--
Robert G. Brown Phone(cell): 1-919-280-8443
Duke University Physics Dept, Box 90305
Durham, N.C. 27708-0305
Web: http://www.phy.duke.edu/~rgb
Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php
Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977
John Leidel
2008-09-16 21:12:55 UTC
Permalink
and, a selfish plug:
http://insidehpc.com/2008/09/16/cray-announces-mini-supercomputer-line/
Post by Gus Correa
Dear Beowulf and COTS fans
http://www.theregister.co.uk/2008/09/16/cray_baby_super/
IGIDH (I guess it doesn't help.)
Gus Correa
David Mathog
2008-09-16 23:01:49 UTC
Permalink
Post by Prentice Bisbal
"It's also attempting to lure scientists and researchers with
discretionary IT budgets to forget using shared, giant clusters and get
their own box and tuck it in behind their desk where no one can see it
to run their workloads locally. The personal supercomputer is not a new
idea, but this is the first time that Cray is trying it out in the
market."
Post by Prentice Bisbal
That will work great until the newbie scientists find that airflow into
a computer tucked in "behind their desk where no one can see it" is piss
poor, and that fans powerful enough to provide adequate airflow "behind
the desk where no one can see it" are going to be LOUD.
Well, they _might_ be able to muffle the air flow noise with an
expensive, and most likely large, case design. The one thing they can't
get away from is the heat. Anything "super", meaning much faster than
an average desktop, is going to use a lot of power, and 1000W is roughly
where I would imagine this sort of machine would sit, the upper limit
being set by the need to leave at least 500W for incidentals like small
printers, phones, displays and the like, on the same 15A circuit. 1000W
is too much heat to have under ones desk, unless a dramatically lowered
sperm count is the goal. It is also pushing the edge of the AC capacity
for a small office. On the other hand, if Cray can make it reasonably
quiet it should be ok in a lab environment, even at 1000W. Labs have
better air flow, higher AC capacity, and better power (multiple 20A
circuits are common) than do offices. Labs are better, but not THAT
much better, so if everybody in the lab wants one, you might be back to
needing a machine room, at which point the expensive quiet case was
a waste of money. Or worse, they might not fit in racks.

Regards,

David Mathog
***@caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech
Lux, James P
2008-09-17 13:22:58 UTC
Permalink
Post by Lux, James P
There is a huge psychological advantage to having the computer
physically under your management and control. You don't have folks
trying to "optimize the use of a valuable institutional resource"
with scheduling, etc. You might be willing to tolerate a factor of
2 hit in performance for the ability to not have to account for
anyone else about how much you're using or not using it.
And then they all expect the central systems support group to get it
running for them, and to fix it when it breaks, and to generally
maintain it. Suddenly you have dozens of completely different systems
scattered far and wide across your site, and you're starting to get
complaints that the support group are unobtainable these days -
they're never at their desk any more, and don't seem to have any time
to build new stuff any more.

Tim


But how is that any different than having a PC on your desk?

I see the deskside supercomputer as a revisiting of the "workstation" class computer. Used to be that PCs and Apples were what sat on most peoples desks, but some had Apollo or Sun or Perq workstations, because they had applications that needed the computational horsepower (or, more likely, the high res hardware graphics support.. A CGA was pretty painful for doing PC board layout).

Same sort of thing for having the old Tektronix 4014 graphics terminal, rather than hiking down to the computer center to pick up your flatbed plotter output.

Jim
John Leidel
2008-09-17 13:45:24 UTC
Permalink
I almost hate to throw this one out there, but does anyone remember the
SGI deskside series? Challenge, Origin, Onyx....

These were fairly popular there in the mid to late 90's. We had one at
GFDL up until at least a year ago.

[I want to score one for my house to play with]
Post by Prentice Bisbal
Post by Lux, James P
There is a huge psychological advantage to having the
computer
Post by Lux, James P
physically under your management and control. You don't
have folks
Post by Lux, James P
trying to "optimize the use of a valuable institutional
resource"
Post by Lux, James P
with scheduling, etc. You might be willing to tolerate a
factor of
Post by Lux, James P
2 hit in performance for the ability to not have to account
for
Post by Lux, James P
anyone else about how much you're using or not using it.
And then they all expect the central systems support group to get it
running for them, and to fix it when it breaks, and to
generally
maintain it. Suddenly you have dozens of completely different systems
scattered far and wide across your site, and you're starting to get
complaints that the support group are unobtainable these days -
they're never at their desk any more, and don't seem to have any time
to build new stuff any more.
Tim
But how is that any different than having a PC on your desk?
I see the deskside supercomputer as a revisiting of the
“workstation” class computer. Used to be that PCs and Apples
were what sat on most peoples desks, but some had Apollo or
Sun or Perq workstations, because they had applications that
needed the computational horsepower (or, more likely, the high
res hardware graphics support.. A CGA was pretty painful for
doing PC board layout).
Same sort of thing for having the old Tektronix 4014 graphics
terminal, rather than hiking down to the computer center to
pick up your flatbed plotter output.
Jim
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
Huw Lynes
2008-09-17 13:50:53 UTC
Permalink
Post by John Leidel
I almost hate to throw this one out there, but does anyone remember the
SGI deskside series? Challenge, Origin, Onyx....
Having experience of all three, I suggest that it's a bit of a stretch
to refer to any of those as "deskside".

I'm just old enough to have had to deal with a Challenge in production.
I still occasionally have to go and prod the old Origins and Onyx in
Computer Science.

Thanks,
Huw
--
Huw Lynes | Advanced Research Computing
HEC Sysadmin | Cardiff University
| Redwood Building,
Tel: +44 (0) 29208 70626 | King Edward VII Avenue, CF10 3NB
Tom Elken
2008-09-17 15:39:10 UTC
Permalink
Post by Alan Louis Scheinine
Post by John Leidel
I almost hate to throw this one out there, but does anyone
remember the
Post by John Leidel
SGI deskside series? Challenge, Origin, Onyx....
Having experience of all three, I suggest that it's a bit of a stretch
to refer to any of those as "deskside".
"Challenge" covered a lot of system types. Some were proper desksides,
but not proper Challenges:
Tim Cutts
2008-09-17 13:51:34 UTC
Permalink
Post by Lux, James P
But how is that any different than having a PC on your desk?
I see the deskside supercomputer as a revisiting of the
"workstation" class computer. Used to be that PCs and Apples were
what sat on most peoples desks, but some had Apollo or Sun or Perq
workstations, because they had applications that needed the
computational horsepower (or, more likely, the high res hardware
graphics support.. A CGA was pretty painful for doing PC board
layout).
Same sort of thing for having the old Tektronix 4014 graphics
terminal, rather than hiking down to the computer center to pick up
your flatbed plotter output.
Jim
We don't generally allow people here to buy their own PCs and Apples
either. They get a standard build from us, all centrally managed by
LanDESK. They also get a known type of hardware; they can't just buy
what the hell they like. I have more than 800 Windows desktops to
support. If they were all different and purchased ad-hoc by
individual users, I would be in even worse hell than I am already.

Most people don't build Beowulf clusters out of ad-hoc piles of
machines from God-knows-where. Most of us buy consistent hardware,
because it's impossible to support anything else.

The Tektronix graphics terminal is slightly different, because it was
just that, a terminal, and consequently doesn't present such a headache.

Tim
--
The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
Lux, James P
2008-09-17 15:51:35 UTC
Permalink
-----Original Message-----
From: Tim Cutts [mailto:***@sanger.ac.uk]
Sent: Wednesday, September 17, 2008 6:52 AM
To: Lux, James P
Cc: Prentice Bisbal; Beowulf
Subject: Re: [Beowulf] MS Cray
Post by Lux, James P
But how is that any different than having a PC on your desk?
I see the deskside supercomputer as a revisiting of the "workstation"
class computer. Used to be that PCs and Apples were what sat on most
peoples desks, but some had Apollo or Sun or Perq workstations,
because they had applications that needed the computational horsepower
(or, more likely, the high res hardware graphics support.. A CGA was
pretty painful for doing PC board layout).
Same sort of thing for having the old Tektronix 4014 graphics
terminal, rather than hiking down to the computer center to pick up
your flatbed plotter output.
Jim
We don't generally allow people here to buy their own PCs and Apples either. They get a standard build from us, all centrally managed by LanDESK. They also get a known type of hardware; they can't just buy what the hell they like. I have more than 800 Windows desktops to support. If they were all different and purchased ad-hoc by individual users, I would be in even worse hell than I am already.

Most people don't build Beowulf clusters out of ad-hoc piles of machines from God-knows-where. Most of us buy consistent hardware, because it's impossible to support anything else.

The Tektronix graphics terminal is slightly different, because it was just that, a terminal, and consequently doesn't present such a headache.

Tim

----

Indeed, and such is the case in most large organizations. Two that I have direct familiarity with have slightly different models. One, in a Fortune 500 company, had, at any time, only 3 possible hardware configurations for the desktop (with literally 10s of thousands deployed), with the actual image rolled out every day. Essentially, the disk drive in the box served as a local cache. There were other configurations for software developers, but still, pretty much locked down. The server farms are run separately, by a centralized organization, as is the mainframe. A small "departmental server" (e.g. for a software development group to use for testing) would be in a server room somewhere, managed by the central org.




The other, here at JPL, has about 10,000 or so computers of various ages and configurations that are managed collectively (as opposed to those being Sysadmined locally,e.g. in a lab). At any given time, there's a dozen or so kinds of computers (desktop/laptop/PC/Mac) available, but since the configurations are changing, and they have a 3 year recycle time, there's probably 30 or 40 configurations in the field at a given time. The software configuration is substantially more consistent, in that there's a basic "core software" load of OS, tools (Office, Mail, Calendaring), but people, in general, have admin access to their own machine, and are free to install anything else (as long as it's legal). OTOH, if something you add causes problems, they're not on the hook to support it, and ultimately, their response might be to reimage the disk. They ARE pushing towards a thin client model, at least for non-specialized desktop users (e.g. if all you do is email, calendaring, documents, and web service consuming). Interestingly, the monthly cost for both organizations is about the same ( a few hundred bucks a month for hardware lease+service). We also have "servers for rent" (with SA and 24/7 monitoring done by others), as well as various and sundry supercomputers. A deskside supercomputer would fit in the model here fairly well, as just another flavor of either high performance desktop machine, or as a small server in your lab.


Jim
Tony Travis
2008-09-17 21:00:11 UTC
Permalink
Post by Tim Cutts
Post by Lux, James P
But how is that any different than having a PC on your desk?
I see the deskside supercomputer as a revisiting of the "workstation"
class computer. Used to be that PCs and Apples were what sat on most
peoples desks, but some had Apollo or Sun or Perq workstations,
because they had applications that needed the computational horsepower
(or, more likely, the high res hardware graphics support.. A CGA was
pretty painful for doing PC board layout).
Same sort of thing for having the old Tektronix 4014 graphics
terminal, rather than hiking down to the computer center to pick up
your flatbed plotter output.
Hello, Jim and Tim.

I agree with you, Jim - I used to do a lot of image analysis, and I had
a 'Torch' VME 68030 workstation on my desk, followed by a Sun SPARC10...
Post by Tim Cutts
We don't generally allow people here to buy their own PCs and Apples
either. They get a standard build from us, all centrally managed by
LanDESK. They also get a known type of hardware; they can't just buy
what the hell they like. I have more than 800 Windows desktops to
support. If they were all different and purchased ad-hoc by individual
users, I would be in even worse hell than I am already.
Poacher turned gamekeeper, Tim?

If you want a 'standard' locked-down desktop go to PC hell. If you
don't, then support it yourself - What's the problem?
Post by Tim Cutts
Most people don't build Beowulf clusters out of ad-hoc piles of machines
from God-knows-where. Most of us buy consistent hardware, because it's
impossible to support anything else.
No, some of us are still part of the rebel alliance - building Beowulf
clusters out of anything we can beg, borrow or steal to do the science
that we are interested in. Were those DEC Alpha's you had 'consistent'
hardware, or did you just buy a new generation of IBM blade servers to
replace them after you decided to buy 'consistent' hardware?

I've got all sorts of kit ranging from Athlon XP 2400+ to Opteron 2212's
in our Beowulf cluster and they are all doing a worthwhile job. I doubt
that my situation is unique. As the saying goes "It's not having what
you want that matters, but wanting what you have..." ;-)
Post by Tim Cutts
The Tektronix graphics terminal is slightly different, because it was
just that, a terminal, and consequently doesn't present such a headache.
You know what really makes my blood boil is being criticised for using
'non-standard' PC's, Mac's... whatever because "it can't be supported",
when I have never wanted or needed support! I'm all in favour of people
who DO want a 'managed' desktop having a standard-issue PC running the
evil empire's wares, but ONLY if that's what the end user wants. Yes,
I've done PC support and I do have experience of the issues involved.

So, back to the point, I'm very much in favour of the 'democratisation'
of HPC as manifested by DIY Beowulf clusters and affordable desk-side
horse-power. HPC on BIG-IRON is not under the control of scientists who
want to use it. That's why they use 'discretionary' budgets to buy the
sort of kit Orion tried to sell, Apple do sell and Cray are pushing onto
the 'personal' HPC market. Indeed, Moshe Bar was so convinced that this
was the way to go that he ended the openMosix project [tears].

Hmm... back to earth now - needed to get that off my chest :-)

Tony.
--
Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition
and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK
tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk
mailto:***@rri.sari.ac.uk, http://bioinformatics.rri.sari.ac.uk/~ajt
Mike Davis
2008-09-17 14:06:08 UTC
Permalink
Post by Lux, James P
But how is that any different than having a PC on your desk?
I see the deskside supercomputer as a revisiting of the
“workstation” class computer. Used to be that PCs and Apples were
what sat on most peoples desks, but some had Apollo or Sun or Perq
workstations, because they had applications that needed the
computational horsepower (or, more likely, the high res hardware
graphics support.. A CGA was pretty painful for doing PC board
layout).
Same sort of thing for having the old Tektronix 4014 graphics
terminal, rather than hiking down to the computer center to pick
up your flatbed plotter output.
Jim
Jim,

One big difference is that this machine will be sold to department
chairs and Deans not as a desktop or high end workstation but as a cheap
"Supercomputer" that needs no support. The PC support available in an
organization may be completely unable to deal with the realities of HPC.
That's when I get an urgent call about a machine that I don't know
about. The clock started ticking the day that the machine arrived and
that's the impossible timetable to which I will be held. In other words,
even with my absolute full attention my efforts will be presented as
failing to set up the machine in a timely manner. In addition all of the
other researchers who have invested in the centralized resources will
complain that they are not getting the attention that they need.

I think that there are times that machines such as this on a
departmental or even researcher level make sense even in an organization
that provides central resources. But those times are the exceptions. I
have 2.5 System Admins. I have ~300 machines in two different locations
as standalone servers and parts of clusters. We can get by with this
level of staffing through standardization of hardware and operating
systems (currently 90% linux, 9% Solaris, 1% IRIX), security standards
that lock down unused ports and services, and careful testing of
software (physical sciences, math, OR, bioinformatics) before it is made
generally available. With budget cuts looming on the horizon, adding
support for new department level systems without additional staffing
would leave us unable to continue to provide adequate support for the
central systems. IMHO. YMMV.
--
Mike Davis Technical Director
(804) 828-3885 Center for High Performance Computing
***@vcu.edu Virginia Commonwealth University

"Never tell people how to do things. Tell them what to do and they will surprise you with their ingenuity." George S. Patton
Lux, James P
2008-09-17 16:03:43 UTC
Permalink
Post by Lux, James P
But how is that any different than having a PC on your desk?
I see the deskside supercomputer as a revisiting of the
"workstation" class computer. Used to be that PCs and Apples were
what sat on most peoples desks, but some had Apollo or Sun or Perq
workstations, because they had applications that needed the
computational horsepower (or, more likely, the high res hardware
graphics support.. A CGA was pretty painful for doing PC board
layout).
Same sort of thing for having the old Tektronix 4014 graphics
terminal, rather than hiking down to the computer center to pick
up your flatbed plotter output.
Jim
Jim,

One big difference is that this machine will be sold to department chairs and Deans not as a desktop or high end workstation but as a cheap "Supercomputer" that needs no support. The PC support available in an organization may be completely unable to deal with the realities of HPC.


--> then the seller is lying about no support required, and the buyer deserves what they get. The real question is whether the box is more like a "super desktop PC" or a "supercomputer".. The former, by design, should have a fairly low admin overhead (i.e. the hardware configuration is fixed and stable, the OS ditto, relatively few applications on it).. The latter has a high SA overhead, because, by it's nature, it's used by a heterogenous group of folks running heterogenous applications which were all developed to stretch the limits. It's whether the new deskside box exhibits "the realities of HPC", or it's just a faster computer like the other ones.

That's when I get an urgent call about a machine that I don't know about. The clock started ticking the day that the machine arrived and that's the impossible timetable to which I will be held. In other words, even with my absolute full attention my efforts will be presented as failing to set up the machine in a timely manner. In addition all of the other researchers who have invested in the centralized resources will complain that they are not getting the attention that they need.

--> That's more of an education and management of expectations.

I think that there are times that machines such as this on a departmental or even researcher level make sense even in an organization that provides central resources. But those times are the exceptions. I have 2.5 System Admins. I have ~300 machines in two different locations as standalone servers and parts of clusters.

---> then you're grossly underfunded and/or your institution is getting a great deal because you and your staff is making a herculean effort. Typical support costs in industry run about $200-300/month per desktop (and that's for fairly vanilla installations).. 300*300 = $90K/mo = $1080K/yr -> 4-6 people.


We can get by with this level of staffing through standardization of hardware and operating systems (currently 90% linux, 9% Solaris, 1% IRIX), security standards that lock down unused ports and services, and careful testing of software (physical sciences, math, OR, bioinformatics) before it is made generally available. With budget cuts looming on the horizon, adding support for new department level systems without additional staffing would leave us unable to continue to provide adequate support for the central systems. IMHO. YMMV.


-->> exactly.. Your operation is on the ragged edge of resources, so your organization really can't tolerate dropping in a new and different sort of box, at least within the desktop PC support model. But for an organization that already has, say, 10K machines, and the staff corresponding to the $30-40M/yr budget (e.g. a hundred people), adding a new flavor of box isn't as disruptive. One of the horde can be detailed off to become the "new widget expert".

--> So perhaps your institution isn't really the appropriate target market (yet).. I don't see this as particularly different than any other new technology introduction. When mainframes first entered the halls of academe, I'm sure the same sort of discussions arose. Heck, it's why computers like the PDP-8 were invented.

Jim
John Hearns
2008-09-17 16:39:52 UTC
Permalink
Post by Lux, James P
. When mainframes first entered the halls of academe, I'm sure the same
sort of discussions arose. Heck, it's why computers like the PDP-8 were
invented.
Jim
Just let me correct you there. Surely PDP-8s were calculators or Data
Processing whatchamacallits,
and emphatically NOT Computer Systems.
(A history lesson is called for here - I cannot remember the exact
terminology which allowed PDPs to be sold to individual labs and
departments)
Lux, James P
2008-09-17 17:18:54 UTC
Permalink
From: John Hearns [mailto:***@googlemail.com]
Sent: Wednesday, September 17, 2008 9:40 AM
To: Lux, James P; ***@beowulf.org
Subject: Re: [Beowulf] MS Cray
. When mainframes first entered the halls of academe, I'm sure the same sort of discussions arose. Heck, it's why computers like the PDP-8 were invented.

Jim



Just let me correct you there. Surely PDP-8s were calculators or Data Processing whatchamacallits,
and emphatically NOT Computer Systems.
(A history lesson is called for here - I cannot remember the exact terminology which allowed PDPs to be sold to individual labs and departments)


--Sort of like when we buy PCs for the lab, they're "instrument controllers"..

Jim
Tony Travis
2008-09-17 21:09:48 UTC
Permalink
Post by John Hearns
[...]
Just let me correct you there. Surely PDP-8s were calculators or Data
Processing whatchamacallits,
and emphatically NOT Computer Systems.
(A history lesson is called for here - I cannot remember the exact
terminology which allowed PDPs to be sold to individual labs and
departments)
Hello, John.

It's interesting to read about the history of PDP's in this context:

http://en.wikipedia.org/wiki/Programmed_Data_Processor

I'm a verteran of Version 7 Unix on the pdp11/23 and 11/34 myself, and I
had an 11/23 in my front room until I bought a PC to replace it :-)

Tony.
--
Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition
and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK
tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk
mailto:***@rri.sari.ac.uk, http://bioinformatics.rri.sari.ac.uk/~ajt
David Mathog
2008-09-17 17:39:30 UTC
Permalink
Post by Lux, James P
The other, here at JPL,
I have heard about this contract before - and in my opinion, it is a
horrible deal. The taxpayers get reamed and the vendor makes out like a
bandit.

(SNIP)
Post by Lux, James P
At any given time, there's a dozen or so kinds of computers
(desktop/laptop/PC/Mac) available, but since the configurations are
changing, and they have a 3 year recycle time,
the key point being the 3 year lease.

(SNIP)
Post by Lux, James P
Interestingly, the monthly cost for both organizations is about the
same ( a few hundred bucks a month for hardware lease+service).
Let "a few hundred" = $200, and of course there are 36 months in 3
years, so JPL pays the vendor $7200 for each machine, plus "support" for
this term. At the end of the lease the vendor gets the computer back,
and they probably sell it for a few hundred dollars, just to sweeten the
already cushy deal. The office staff may need support once and a while,
but one can assume that the average JPL engineer or scientist can more
than handle all their own PC software issues, and at worst would just
need to swap a machine if there was a major hardware issue. In other
words, the average support cost to the vendor for the technical staff is
but a tiny fraction of what JPL pays them.

Who's the vendor, Halliburton?

Regards

David Mathog
***@caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech
Lux, James P
2008-09-17 18:39:58 UTC
Permalink
Post by Lux, James P
-----Original Message-----
Sent: Wednesday, September 17, 2008 10:40 AM
Cc: Lux, James P
Subject: Re: MS Cray
Post by Lux, James P
The other, here at JPL,
I have heard about this contract before - and in my opinion,
it is a horrible deal. The taxpayers get reamed and the
vendor makes out like a bandit.
actually not.. It's quite competitive with the sort of deal other folks get in industry.
Post by Lux, James P
(SNIP)
Post by Lux, James P
At any given time, there's a dozen or so kinds of computers
(desktop/laptop/PC/Mac) available, but since the configurations are
changing, and they have a 3 year recycle time,
the key point being the 3 year lease.
(SNIP)
Well.. That's always a trade.. Buying comes out of the capital bucket, leasing comes out of the expense bucket, and they have very different treatments, accounting wise. Don't forget that JPL works on a "cost reimbursement" basis: that is, we spend money, and the gov't, via NASA, reimburses those costs, and ONLY those costs. There are literally bookshelves full of rules (the Federal Acquisition Regulations) that tell you what is allocable, accountable, and reimburseable. JPL cannot borrow money to pay for things: the US government will not pay for interest on borrowed money. Most capital investments have to be approved by an act of congress, and the amortization of that investment requires special treatment to make sure that costs are properly allocated to each project. Leasing makes it easy..Accountants cost money too, if you do march down the amortization process. Many, many commercial companies have similar sorts of issues, particularly with respect to transactions between divisions (e.g. see Regulation W for banking), all designed to prevent "hiding profits".
Post by Lux, James P
Post by Lux, James P
Interestingly, the monthly cost for both organizations is about the
same ( a few hundred bucks a month for hardware lease+service).
Let "a few hundred" = $200, and of course there are 36 months
in 3 years, so JPL pays the vendor $7200 for each machine,
plus "support" for this term. At the end of the lease the
vendor gets the computer back, and they probably sell it for
a few hundred dollars, just to sweeten the already cushy
deal.
actually, a mid range desktop machine is about $150, and they actually do monthly comparisons against other vendors, so the cost is probably pretty reasonable for a large volume consumer. Don't forget that part of the cost is the (legally required) paperwork to prove that the taxpayer's not getting the shaft. (e.g. those comparisons don't happen for free, and someone has to collate the data, put it into a report, etc.)


The office staff may need support once and a while, but one can assume that the average JPL engineer or scientist can more than handle all their own PC software issues, and at worst would just need to swap a machine if there was a major hardware issue. In other words, the average support cost to the vendor for the technical staff is but a tiny fraction of what JPL pays them.



The office staff may need support once and a while,
Post by Lux, James P
but one can assume that the average JPL engineer or scientist
can more than handle all their own PC software issues, and at
worst would just need to swap a machine if there was a major
hardware issue. In other words, the average support cost to
the vendor for the technical staff is but a tiny fraction of
what JPL pays them.
Actually, before they went to a centralized support model, they found that the "shadow administrator" and "shadow support staff" costs were huge, and more to the point, not accurately measureable (what you can't measure, you can't manage). And, while most JPL technical staff are certainly capable of doing their own support, there's good reason for them not to: they typically have a more than full time job doing something more specialized (e.g. designing deep space communication links, or writing nav software for rovers, or designing and building rovers), and every minute they spend fooling with their computer is a minute not getting the *real job* of space exploration done. It's like working on my house: I'm capable of painting the walls myself, but I hire a professional to do it, because there are better uses for my time (and the specialist will do a better job).

Support isn't trivial here, even for office staff. For a variety of reasons (and not unique to JPL.. Any other 5000+ employee, $1B year business will have similar ones), we have an amazingly wide array of various and sundry institutional applications to do things like timecards, keeping track of inventory, document mangement, etc. I'd venture that the desktop support staff spends more than 70% of their time dealing with non-OS related software issues (e.g. why is my email not getting through), and a very tiny fraction of their time responding to hardware problems or OS issues. For instance, they're rolling out a new "unified messaging system" which will integrate calendaring, based on MS Exchange Server (for whatever reason.. That's irrelevant), and rather than saying "Use Outlook", which would certainly reduce support costs, they ARE supporting half a dozen email clients, web clients, a bunch of calendaring clients, not to mention a fairly heterogenous set of hardware platforms.

Let us not forget IT security. This is a non-trivial matter when you have to manage tens of thousands of desktops and comply with dozens of pages of government regulation, NASA procedural instructions, etc.

The upshot is, you get a fair amount for your $130/month support subscription. To put that in context, that's less than two hours of engineer time.
Post by Lux, James P
Who's the vendor, Halliburton?
No, it's a division of Lockheed Martin, but basically anyone in the business of doing what they do charges about the same amount of money. The business *is* fairly competitive (and gets recompeted every 3-5 years, I think), and they changed vendors a few cycles ago.

For what it's worth, we have a similar sort of scenario when dealing with test equipment. Do you own it, and have inhouse inventory, (with all the peculiar government contract stuff about cost accounting), or do you have an outside vendor provide it on lease/rent. Same sorts of heterogenous product line. Same sorts of issues with support (e.g. test and calibration of instruments). Same sort of problems with purchasing capital equipment on a cost reimbursement job (the government discourages you charging the full cost of your infrastructure to their job).
A big disadvantage of in-house is that it leads to an inventory of ancient gear that becomes hard to maintain, and balkanized ownership (we bought that for Project X, and though Project X is long gone, the former staff of Project X still owns it). A big advantage of in-house is that you can walk across the street and pick up an oscilloscope.

Jim
Lux, James P
2008-09-22 23:52:52 UTC
Permalink
Post by Lux, James P
-----Original Message-----
Sent: Monday, September 22, 2008 4:13 PM
To: Lux, James P
Subject: Re: [Beowulf] RE: MS Cray
Post by Lux, James P
A big disadvantage of in-house is that it leads to an inventory of
ancient gear that becomes hard to maintain, and balkanized
ownership
Post by Lux, James P
(we bought that for Project X, and though Project X is long
gone, the
Post by Lux, James P
former staff of Project X still owns it). A big advantage
of in- house
Post by Lux, James P
is that you can walk across the street and pick up an oscilloscope.
Jim
We had some issues with these NASA owned machines in the
previous department i worked for. Because of arcane rules of
ownership and what could be done with these machines, they
sat in a closet for years (they may still be there) because
noone knew what could be legally done with these extrememly
outdated and useless machines.
Naveed
And with NASA exploration missions, you may need to keep ancient gear "under glass" for a long time, to support some spacecraft that was launched 30 years ago (Voyager 1&2 are still going strong, eh?). Some anomaly occurs, you need to drag out the testbed and the test equipment that was connected to the test bed, etc. Maybe all your test data is stored on diskettes that were written by the Apple II that was the instrument controller.


James Lux, P.E.
Task Manager, SOMD Software Defined Radios
Flight Communications Systems Section
Jet Propulsion Laboratory
4800 Oak Grove Drive, Mail Stop 161-213
Pasadena, CA, 91109
+1(818)354-2075 phone
+1(818)393-6875 fax
Naveed Near-Ansari
2008-09-22 23:13:05 UTC
Permalink
Post by Lux, James P
A big disadvantage of in-house is that it leads to an inventory of
ancient gear that becomes hard to maintain, and balkanized ownership
(we bought that for Project X, and though Project X is long gone,
the former staff of Project X still owns it). A big advantage of in-
house is that you can walk across the street and pick up an
oscilloscope.
Jim
We had some issues with these NASA owned machines in the previous
department i worked for. Because of arcane rules of ownership and what
could be done with these machines, they sat in a closet for years
(they may still be there) because noone knew what could be legally
done with these extrememly outdated and useless machines.

Naveed
Nifty niftyompi Mitch
2008-09-29 04:44:49 UTC
Permalink
Post by Naveed Near-Ansari
Post by Lux, James P
A big disadvantage of in-house is that it leads to an inventory of
ancient gear that becomes hard to maintain, and balkanized ownership
(we bought that for Project X, and though Project X is long gone, the
former staff of Project X still owns it). A big advantage of in-house
is that you can walk across the street and pick up an oscilloscope.
Jim
We had some issues with these NASA owned machines in the previous
department i worked for. Because of arcane rules of ownership and what
could be done with these machines, they sat in a closet for years (they
may still be there) because noone knew what could be legally done with
these extrememly outdated and useless machines.
Interesting... If I recall my history correctly Unix was first prototyped on a cast off machine...
--
T o m M i t c h e l l
Found me a new hat, now what?
Robert G. Brown
2008-09-17 19:08:16 UTC
Permalink
Post by David Mathog
Let "a few hundred" = $200, and of course there are 36 months in 3
years, so JPL pays the vendor $7200 for each machine, plus "support" for
this term. At the end of the lease the vendor gets the computer back,
and they probably sell it for a few hundred dollars, just to sweeten the
already cushy deal. The office staff may need support once and a while,
but one can assume that the average JPL engineer or scientist can more
than handle all their own PC software issues, and at worst would just
need to swap a machine if there was a major hardware issue. In other
words, the average support cost to the vendor for the technical staff is
but a tiny fraction of what JPL pays them.
Who's the vendor, Halliburton?
Brilliantly put. And yet people buy in! Why?

In a typical small business, they may well have no real systems
administrators. "Administration" may be done by an office manager, a
secretary, a staff person with a bit of computing in their blood or
better than average common sense. In such an environment, the (say)
$4200 surplus times 30 or 50 machines may end up being cheaper than
hiring a full time real sysadmin, although this is an extreme example
where one is right on the margin with these particular numbers even for
that on the 50 end of things. Add to that, though, that Windows admin
isn't terrribly scalable and things break a lot, so one admin cannot
handle linux-like numbers of boxes, so it is still not quite crazy.

The real question is why an admin-rich environment with lots of full
time admins would ever buy into such a deal. If you've got a full time
admin ANYWAY, paying $150/month for support on top of this (beyond the
cost of the hardware is just insane.

rgb
Post by David Mathog
Regards
David Mathog
Manager, Sequence Analysis Facility, Biology Division, Caltech
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
--
Robert G. Brown Phone(cell): 1-919-280-8443
Duke University Physics Dept, Box 90305
Durham, N.C. 27708-0305
Web: http://www.phy.duke.edu/~rgb
Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php
Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977
Joe Landman
2008-09-17 19:54:29 UTC
Permalink
Post by Robert G. Brown
The real question is why an admin-rich environment with lots of full
time admins would ever buy into such a deal. If you've got a full time
admin ANYWAY, paying $150/month for support on top of this (beyond the
cost of the hardware is just insane.
Have you ever administered a lab full of these units? You need as much
help as you can get to administer the windows machines. Sadly, while
claims of there being more windows admins are true (thats not the sad
part) you need (far) more to administer fewer windows machines than the
fewer admins needed for more Linux machines (that is the sad part).

We have seen 2 full time admins handle 4000+ Linux machines with time to
develop software to make their lives easier (Incyte), as compared to
seeing 10 windows admins struggle to keep 100 machines each up to date.

The cluster version of w2k* should alleviate or at least make it
tolerable for clusters. Desktops are another matter. Part of the
reason that it is locked down so tight in large organizations is that
when it fails (not "if") they want to reduce the degrees of freedom of
the failure modes. Keep it simple, in their own way.

But that is straying a bit. Large organizations often are not admin
rich, they tend to try to cut out costs. Admins == costs. Admins need
as much help as they can possibly get. If for $150/machine in a large
organization, you can take away some level of their pain, this might be
worth more than the cost of the additional healthcare coverage,
heartburn medicine, and upper/lower GI series needed ...
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: ***@scalableinformatics.com
web : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax : +1 734 786 8452
cell : +1 734 612 4615
Lux, James P
2008-09-17 20:27:55 UTC
Permalink
Post by Lux, James P
-----Original Message-----
Sent: Wednesday, September 17, 2008 12:54 PM
To: Robert G. Brown
Subject: Re: [Beowulf] Re: MS Cray
Post by Robert G. Brown
The real question is why an admin-rich environment with
lots of full
Post by Robert G. Brown
time admins would ever buy into such a deal. If you've got a full
time admin ANYWAY, paying $150/month for support on top of this
(beyond the cost of the hardware is just insane.
Have you ever administered a lab full of these units? You
need as much help as you can get to administer the windows
machines. Sadly, while claims of there being more windows
admins are true (thats not the sad
part) you need (far) more to administer fewer windows
machines than the fewer admins needed for more Linux machines
(that is the sad part).
We have seen 2 full time admins handle 4000+ Linux machines
with time to develop software to make their lives easier
(Incyte), as compared to seeing 10 windows admins struggle to
keep 100 machines each up to date.
I think part of the problem in the Windows world is the incredible diversity of applications (by which I include websites with significant client side processing) that wind up being run on them. Rich growth medium, lots of spontaneous mutations.

When you get to large desktop rollouts, Windows can have fairly low admin overhead, but it's done by restricting flexibility (e.g. SMS, boot from the network, etc.) to reduce the nutritional value of the growth medium. If everyone boots the same image from the net, applying a patch to 10,000 computers is trivial. While such an environment would probably make everyone on this list exceedingly unhappy (I could guarantee there's no compiler of any kind in it..you might get a JRE, and edit your source code in MSWord), it would (and does) serve a huge number of folks in the business world perfectly well.

Windows in a development intensive, HPC environment, is going to be admin expensive.

Jim
Joe Landman
2008-09-17 20:35:22 UTC
Permalink
Post by Lux, James P
When you get to large desktop rollouts, Windows can have fairly low
admin overhead, but it's done by restricting flexibility (e.g. SMS,
boot from the network, etc.) to reduce the nutritional value of the
Sadly, I haven't seen this at the large customer rollouts I have seen.
It is anything but trivial. In any respect of this word.

Some of the organizations I have seen have now, completely thrown their
hands in the air over it, and hand admin rights on the laptops back to
the users. They tell them what they can and cannot break, and then let
them install/manage their own software (apart from corporate software
pushes).

Remarkably, this appears to be reducing cost/time/headache on admin. As
this is a fortune 100 company of which I speak, I am sure they are not
alone. Someones workstation gets hosed, they re-image from a known
start. Off they go again. Their data is their responsibility.

Interesting to see how behavior changes then.

[...]
Post by Lux, James P
Windows in a development intensive, HPC environment, is going to be admin expensive.
Hopefully future windows will look a great deal more like w2k8, so
hopefully the pain will be lower.

If I can get a reasonable cost license for w2k8, I could easily see
running it on my laptop (over XP).

Joe
Post by Lux, James P
Jim
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: ***@scalableinformatics.com
web : http://www.scalableinformatics.com
http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
Tim Cutts
2008-09-18 08:45:33 UTC
Permalink
Post by Joe Landman
Some of the organizations I have seen have now, completely thrown
their hands in the air over it, and hand admin rights on the laptops
back to the users. They tell them what they can and cannot break,
and then let them install/manage their own software (apart from
corporate software pushes).
Laptops are particularly problematic, usually because (a) they don't
connect to the institutional network often enough to acquire updates,
and (b) because the user usually gets given admin rights, as you say,
so that they can make changes that might be required while they're
away at a conference or whatever. They are definitely a major
headache for me...

Tim
--
The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
Eric Thibodeau
2008-09-17 20:52:58 UTC
Permalink
Post by Lux, James P
Post by Lux, James P
-----Original Message-----
Sent: Wednesday, September 17, 2008 12:54 PM
To: Robert G. Brown
Subject: Re: [Beowulf] Re: MS Cray
Post by Robert G. Brown
The real question is why an admin-rich environment with
lots of full
Post by Robert G. Brown
time admins would ever buy into such a deal. If you've got a full
time admin ANYWAY, paying $150/month for support on top of this
(beyond the cost of the hardware is just insane.
Have you ever administered a lab full of these units? You
need as much help as you can get to administer the windows
machines. Sadly, while claims of there being more windows
admins are true (thats not the sad
part) you need (far) more to administer fewer windows
machines than the fewer admins needed for more Linux machines
(that is the sad part).
We have seen 2 full time admins handle 4000+ Linux machines
with time to develop software to make their lives easier
(Incyte), as compared to seeing 10 windows admins struggle to
keep 100 machines each up to date.
I think part of the problem in the Windows world is the incredible diversity of applications (by which I include websites with significant client side processing) that wind up being run on them. Rich growth medium, lots of spontaneous mutations.
When you get to large desktop rollouts, Windows can have fairly low admin overhead, but it's done by restricting flexibility (e.g. SMS, boot from the network, etc.) to reduce the nutritional value of the growth medium. If everyone boots the same image from the net, applying a patch to 10,000 computers is trivial. While such an environment would probably make everyone on this list exceedingly unhappy (I could guarantee there's no compiler of any kind in it..you might get a JRE, and edit your source code in MSWord), it would (and does) serve a huge number of folks in the business world perfectly well.
Nonsense, in that line of thoughts, all that is required is to maintain
a "dev" desktop image to be served through the network. I do it with
linux with quite some ease, I see no reasons why this wouldn't be the
case under windows (meaning that once you're able to boot a network
image under windows, the mechanics of it should be trivially flexible
enough to point to differing images).
Post by Lux, James P
Windows in a development intensive, HPC environment, is going to be admin expensive.
...thinking about all those admins that never wanted to learn Linux, the _will_ have to learn something new this time!

Eric
Robert G. Brown
2008-09-17 22:15:22 UTC
Permalink
Post by Lux, James P
I think part of the problem in the Windows world is the incredible
diversity of applications (by which I include websites with significant
client side processing) that wind up being run on them. Rich growth
medium, lots of spontaneous mutations.
When you get to large desktop rollouts, Windows can have fairly low
admin overhead, but it's done by restricting flexibility (e.g. SMS, boot
from the network, etc.) to reduce the nutritional value of the growth
medium. If everyone boots the same image from the net, applying a patch
to 10,000 computers is trivial. While such an environment would
probably make everyone on this list exceedingly unhappy (I could
guarantee there's no compiler of any kind in it..you might get a JRE,
and edit your source code in MSWord), it would (and does) serve a huge
number of folks in the business world perfectly well.
Sure, but anyone on this list who has ever run a Linux based client
server desktop LAN (which I have, off and on, for around 12 years now,
with Unix LANs in general now up to around 22 years) knows that Linux
beats MS hands down, no questions asked, go home boys on this PROVIDED
ONLY that you choose your hardware sensibly. Linux approaches the
theoretical limits of management scalability -- install your system by
making a single table entry and turning it on (having kindly asked the
vendor to preset the BIOS to boot via DHCP). Then forget it. It
configures up with all user account space set up and ready to roll. It
automatically updates itself. It is far more secure out of the box than
any version of Windows I've ever heard of, and the updates actually
release fixes for critical problems in LESS than six months, usually.

It does require somebody with perhaps more native intelligence than your
typical MCSE -- not to dis on MCSEs, but there it is -- to set this up,
but once set up (which is not that difficult) it takes VERY little time
to actually run. Most Linux admins I know are limited in their work by
the frequency of HARDWARE problems, not software, on a background of
user support. With MS, it tends to be the other way around -- one
spends far more time supporting, tweaking, fixing, rebooting the
software, with about the SAME (or a bit more) user support, and with of
course the same base hardware support. So obviously, you can do a lot
fewer systems.

I don't know that I agree about the application richness of Win compared
to Lin anymore. Lin out of the box comes with far more stuff installed
or immediately available than any Win system, and there is a high cost
to even achieving approximate parity with Lin on the big stuff, --
Office, Web etc. And one can turn a Linux desktop into a Linux server
by going "Poof! You're a server!" For free. In a matter of five
minutes, at any time. yum install apache, plus a bit of this and that.
Been there, done that, totally casually as I decide that I do want my
current laptop to have a development webserver. My sister works in a
serious software devel shop with lots of servers and mainframes and all
(old legacy app stuff) and she is constantly amazed that my laptop is
effortlessly more powerful and better equipped to do ALL the jobs of
both developer and server than most of the hardware and software she
works on and with.

Windows still rules with games. Windows definitely still rules with
certain "mission critical applications" that only run on Windows
(although they aren't EVERYBODY'S mission critical applications). But
for ordinary daily desktop usage in a business setting, Win has long
since fallen BEHIND Lin, with even its edge in hardware drivers and
support steadily dwindling.

And yeah, then there is programming and development and HPC. Here Linux
continues to reign more or less unchallenged, and with the exception of
commercial applications with narrowly targeted but lucrative markets, I
expect that it will remain so.

rgb
Post by Lux, James P
Windows in a development intensive, HPC environment, is going to be admin expensive.
Jim
--
Robert G. Brown Phone(cell): 1-919-280-8443
Duke University Physics Dept, Box 90305
Durham, N.C. 27708-0305
Web: http://www.phy.duke.edu/~rgb
Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php
Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977
Tim Cutts
2008-09-18 08:42:21 UTC
Permalink
Post by Lux, James P
I think part of the problem in the Windows world is the incredible
diversity of applications (by which I include websites with
significant client side processing) that wind up being run on them.
Rich growth medium, lots of spontaneous mutations.
When you get to large desktop rollouts, Windows can have fairly low
admin overhead, but it's done by restricting flexibility (e.g. SMS,
boot from the network, etc.) to reduce the nutritional value of the
growth medium. If everyone boots the same image from the net,
applying a patch to 10,000 computers is trivial. While such an
environment would probably make everyone on this list exceedingly
unhappy (I could guarantee there's no compiler of any kind in
it..you might get a JRE, and edit your source code in MSWord), it
would (and does) serve a huge number of folks in the business world
perfectly well.
Windows in a development intensive, HPC environment, is going to be admin expensive.
LanDESK allows us some considerable flexibility (although it's
expensive software). You can associate particular software packages
with particular users (so that if they log into a machine which
doesn't have that software installed, it happens automatically). You
can set up a self-service portal so that users can install particular
packages which the sysadmin can support centrally, but which don't
have to be installed everywhere.

Admittedly, this does mean that a sysadmin has to package up the
various bits and pieces of software, but this isn't terribly difficult
in most cases. Coupled with the extra benefits that LanDESK provides
(automatic application of security patches to both Windows and many
common Windows applications, reporting of machines which are out of
date, remote control for helping users, etc etc) it's quite a powerful
product.

It does cost, though. We're maintaining about 800 Windows PC's and
about 200 Macs with LanDESK, with one full time admin responsible for
that.

OB Beowulf: LanDESK can also manage SLES and RHAS Linux boxes,
Solaris boxes and AIX, amongst others, although I don't really see the
point - it's fairly easy to automate administration of UNIX systems
anyway.

Tim
--
The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
Robert G. Brown
2008-09-17 21:49:12 UTC
Permalink
Have you ever administered a lab full of these units? You need as much help
as you can get to administer the windows machines. Sadly, while claims of
there being more windows admins are true (thats not the sad part) you need
(far) more to administer fewer windows machines than the fewer admins needed
for more Linux machines (that is the sad part).
There was a lot to parse there, but I think I precisely agreee, and sort
of said that.
We have seen 2 full time admins handle 4000+ Linux machines with time to
develop software to make their lives easier (Incyte), as compared to seeing
10 windows admins struggle to keep 100 machines each up to date.
Yeah, that's precisely our experience here as well.

But somehow, in spite of the fact that hardware is CHEAP and it is
PEOPLE that are expensive, you never see this actually discussed in a
CBA. Microsoft has worked near-miracles of FUD-seeding to convince
people that alas don't know any better that it is somehow the other way
around.
But that is straying a bit. Large organizations often are not admin rich,
they tend to try to cut out costs. Admins == costs. Admins need as much
help as they can possibly get. If for $150/machine in a large organization,
you can take away some level of their pain, this might be worth more than the
cost of the additional healthcare coverage, heartburn medicine, and
upper/lower GI series needed ...
I actually do a bit of consulting -- nothing like what you do -- for an
organization that runs a lot of Windows boxes. It drives me absolutely
nuts. If they were running a set of 40+ plus Linux desktops and a small
stack of Linux-only servers, I could manage the entire operation, close
to 24x7, in even MY copious spare time (not) with time to spare. If
anybody had a problem, I could trivially fix it from home. And they'd
basically never crash or (with 40 clients to 3+servers) EVER EVER load
the network or the servers to the point where I could detect a
meaningful load, given what they are doing. I could be absolutely
certain, when problems occurred, where they occurred and why.

With a mixed lin-win operation (several key linux servers) it is
maddening. The Windows systems crash. They hang, for no reason, even
if they aren't "doing" anything. Unless one is ubervigilant (or in
spite of it) they get viruses. The primary operational server for the
whole LAN is linux, and almost never has a load average over around 0.3,
yet responsiveness drops to near zero in resonance with the primary
Windows server -- that does almost nothing that any human can see that
should be involved with the primary work tool -- peaking in load.

Then there is messing with licensing, buying copies of Office where
needed, installing AV everywhere, getting the printers to work,
configuring the systems to "join a workgroup" and access server-based
disk. Windows is "easy"? Cheap to manage? Fast and efficient in a LAN
client/server context?

I see only two advantages to Windows over Linux these days, and only one
of them matters. The one that matters is there is still a lot of
software that only runs under Windows. Even for the application that is
the core application in this organization, where there actually is a
linux version of the server, the clients only run under Windows. That's
it, right there. The other advantage is that Windows still has a slim
edge in hardware drivers and support -- one is "guaranteed" that Windows
will install on anything you buy and work with all the hardware simply
by virtue of the fact that it comes preinstalled with Windows already
working with it. That is in part why many vendors -- even enlightened
ones that do have linux-based servers for their clients -- still require
windows for the clients. They can be sure that the printers will work,
that (however slow and clunky the configuration) the NIC will function,
they can "borrow" things like MS Word and use them in place of writing
an actual printer stack of their own (yes, people use MS Word as printer
interface/library, believe it or not). They can be sure that their
product will run on NEARLY any hardware package a user comes up with.

Part of even this is FUD, these days. The last few years, Linux has
worked nearly flawlessly out of the box with nearly all desktop
hardware, and is doing pretty decently even on rapidly varying laptop
hardware. Microsoft has once again broken MS Office relative to all
previous versions (while converting to an "open" XML for the docs
themselves) , leveling the playing field with Open Office and providing
it with a surprising opportunity to pull very nearly dead even. Vista
of Evil sucks and everybody knows it even if MS tries to convince them
otherwise with their "Mohave" ads. Apple (running basically Unix)
openly mocks Microsoft on TV and people believe their ads. Cracks --
some of them quite serious -- have appeared in MS's once invincible
facade.

It's hard to say what will happen over the next year or two. I've been
predicting MS's crash-n-burn for years now, and will readily admit I've
been wrong at least about the time frame -- it has taken Linux longer to
get to where it is now than I expected, and the mass of all those WinXX
boxes out there proved more robust and long lived than I expected
(helped considerably by the fact that XPpro, SP2 and then SP3, did not
really suck all that much outside of its absurd price and the long waits
to update through limited bandwidth lines with arcane license validation
tools in the pathway. However, I also remember DEC -- one year it
looked rock-solid, then in the twinkling of a year it failed to make a
hardware cut and in a year more it disappeared without a trace after
decades of phenomenal growth. I remember Apple sinking almost to the
point of bankruptcy. I remember OS/2. There is nothing more fickle and
cruel than the public when the price/performance of something like a
computer OS tips past a certain balance point.

Ubuntu, at long last, has even put desktop linux out there with enough
hardware support and "stuff" that many former Windows users are finding
themselves quite content with it. One of these years RH or some other
big player is going to realize two things:

a) Desktop Linux, sold for 1/3 or 1/6 of the price of any version of
Vista (where the price buys a modest amount of SUPPORT, not the
software), can make them small money over large volume and actually be a
big source of profit.

b) TV ads work. Even as a loss leader, to shake confidence in
Microsoft and to convince people that they can actually run linux on a
desktop and get all their work and a lot of their play done, with less
hassle and expense an ad campaign targeted at consumers is totally worth
it. Look at Apple's success at portraying MSWin as the brain-dead piece
of trash that it is.

Are we at the tipping point? Who knows. MS Windows 7 is supposed to
come to the rescue about now and revert to non-suckiness per desktop,
but Server 2008 and a LAN full of clients still looks nightmarishly
difficult to install, maintain, manage, support. If only users -- or
corporate buyers and administrators -- WOULD factor in the $200/month
that MS thinks is a "bargain" for lease/management per box into the
overall cost of their system, MS might well continue their slow spiral
down into V7. I don't see any killer apps on the horizon, and
"transparency" (at the cost of 4 GB systems and immense slowness) is
obvious hype and glitz without substance. As the marginal difference
between Win and Lin functionality on an installed desktop continues to
shrink, commercial third party applications may soon by MS's ONLY edge,
and I've noted that the commercial 3rd party app aisles for Win continue
to shrink at e.g. Best Buy (except for games). After all, why develop
for Windows, given the CERTAINTY that any killer app you develop and
release will be cloned by MS and that they'll drive you out of business
(or if they feel nice, buy you out of business, their option) at any
moment?

The really good developers work elsewhere, these days, and the
entrepreneurial excitement is drained off into Google, into Amazon, into
open source, into the web (where MS is far from monolithic and where
truly large scale Win-only operations are rather scarce).

Things are dicey. The economy sucks, companies look for ways to cut
corners. Suddenly that enormous pile of Windows licenses, the
one-admin-per-fifty systems looks exorbitant and expensive. Somebody
tells them a Linux admin can do 200+ systems with no license fees (and a
lot more security and fewer user headaches) and the "luxury" of
$200/month Window admin is a real candidate for the block (as in
chopping).

rgb
--
Robert G. Brown Phone(cell): 1-919-280-8443
Duke University Physics Dept, Box 90305
Durham, N.C. 27708-0305
Web: http://www.phy.duke.edu/~rgb
Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php
Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977
Tim Cutts
2008-09-18 08:32:38 UTC
Permalink
Post by Joe Landman
We have seen 2 full time admins handle 4000+ Linux machines with
time to develop software to make their lives easier (Incyte), as
compared to seeing 10 windows admins struggle to keep 100 machines
each up to date.
To be fair -- and I used to work for Incyte, but as a programmer not a
sysadmin -- the sysadmin group was *much* bigger than that. Yes, Stu
started off the Linux clustering as a stealth project between him and
one other guy, whose name I'm ashamed I forget, but you have to
remember that a lot of the routine sysadmin stuff like DNS,
networking, authentication, account management, backups, and whatnot,
were out of their hands, handled by the [really quite large] central
system administration team. By the time they were up to a couple of
thousand nodes, there were definitely more of them running it. For a
start, IIRC there was no PXE provisioning, so it was one unfortunate
sysadmin's job to go round with a crash cart and a boot floppy
whenever a machine needed installing.

Incyte's systems team model was a central team which did all the core
infrastructure stuff, and then individual teams around the company had
dedicated embedded local sysadmins.

As always, my memory is subject to error, so if there are any ex-Palo-
Alto-Incyte sysadmins on the list, feel free to correct me...

Tim
--
The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
Lux, James P
2008-09-17 20:12:33 UTC
Permalink
Post by Lux, James P
-----Original Message-----
Sent: Wednesday, September 17, 2008 12:08 PM
To: David Mathog
Subject: Re: [Beowulf] Re: MS Cray
Post by David Mathog
Let "a few hundred" = $200, and of course there are 36 months in 3
years, so JPL pays the vendor $7200 for each machine, plus
"support"
Post by David Mathog
for this term. At the end of the lease the vendor gets the
computer
Post by David Mathog
back, and they probably sell it for a few hundred dollars, just to
sweeten the already cushy deal. The office staff may need support
once and a while, but one can assume that the average JPL
engineer or
Post by David Mathog
scientist can more than handle all their own PC software
issues, and
Post by David Mathog
at worst would just need to swap a machine if there was a major
hardware issue. In other words, the average support cost to the
vendor for the technical staff is but a tiny fraction of
what JPL pays them.
Post by David Mathog
Who's the vendor, Halliburton?
Brilliantly put. And yet people buy in! Why?
In a typical small business, they may well have no real
systems administrators. "Administration" may be done by an
office manager, a secretary, a staff person with a bit of
computing in their blood or better than average common sense.
In such an environment, the (say) $4200 surplus times 30 or
50 machines may end up being cheaper than hiring a full time
real sysadmin, although this is an extreme example where one
is right on the margin with these particular numbers even for
that on the 50 end of things. Add to that, though, that
Windows admin isn't terrribly scalable and things break a
lot, so one admin cannot handle linux-like numbers of boxes,
so it is still not quite crazy.
The real question is why an admin-rich environment with lots
of full time admins would ever buy into such a deal. If
you've got a full time admin ANYWAY, paying $150/month for
support on top of this (beyond the cost of the hardware is
just insane.
In the model here at JPL, you wouldn't have a full time SA anyway. You'd get the services of one of the instutional SA army for your monthly fee.

If you have a "computing project" (say, building a cluster), and a fulltime SA is part of the project, then you'd not buy the institutional services. Your private SA is then responsible for all the institutional requirements (which can be substantial). Such computers are "unsubscribed" computers, and you have to go through a (fairly simple) justification process to explain how it's not actually a desktop. (The cost to provide "desktop" support within an organization has been documented as being higher than from the outside vendor)

The institutional contract is very much oriented towards "desktop" services, not towards "software development support". We have hundreds of computers that are not subscribed, and which have dedicated SA services, provided by inhouse staff.

I think that rgb is right, though, that there are certain size and structure organizations for which things are different. It's tied to the "granularity" of buying people.

For very small organizations, it's the "office mgr as admin" model, and they might call a consultant in for more complex things (I used to earn my living doing just this.. I'd do the server configs, cabling, etc, set up accounts, etc., build scripts.) I think the threshold for this is when SA duties are down in the few hours/week range.

Then there are very large organizations (like JPL, or Fortune 500 companies) for which standardizing desktops and centralized support make sense.

In between, it's when you have organizations that have enough work to justify about a full time person (or, maybe, 1/2 a person) for their SA job and moving up to a few FTE. You likely have some SA reserve capacity, so the incremental cost for adding one computer is small (as in almost zero), and in this situation, the choice between delta pay for an SA (zero) and $150/month for a service contract is an easy one to make.

Overall institutional structure has a lot to do with it. Academia, in general, has a strong departmental structure, and individual departments tend to be run as independent businesses. As a result, they're sort of working like small/medium sized businesses, with the same zero delta $ for SA vs monthly fee. Some departments are full of technically skilled folks, some of whom for which the incremental hourly cost is very low. If you have someone on salary, "can you spend a half hour looking at my computer" is free (until that poor SA gets overloaded and burns out, or graduates).

And, small entreprenurial organizations are incorrigble optimists. (anyone in academic research has to be an optimist,.. The bold grant proposal typically wins.) They will willingly accept the risk that something that is expensive to fix won't crop up. Very small organizations take the risk, because they can't afford otherwise. More than one small company has gone out of business because their computer died, taking all the business records with it. Very large organizations are conservative.

To a certain extent, the monthly fee is sort of like insurance.. You can gamble that no "big problem" crops up that is beyond your inhouse SA capability, and that you can do minimal SA work. OTOH, you can pay the monthly fee, and when a disaster occurs (a virus infects all your computers, or there's a regulatory compliance edict that requires you to do something to all 50 of your computers), the service provider sucks it up and does it. (on the odds that not everyone has a problem at the same time)


Jim
David Mathog
2008-09-17 20:33:25 UTC
Permalink
Post by Lux, James P
Well.. That's always a trade.. Buying comes out of the capital bucket,
leasing comes out of the expense bucket, and they have very different
treatments, accounting wise. Don't forget that JPL works on a "cost
reimbursement" basis: that is, we spend money, and the gov't, via
NASA, reimburses those costs, and ONLY those costs. There are
literally bookshelves full of rules (the Federal Acquisition
Regulations) that tell you what is allocable, accountable, and
reimburseable.
Translation: "bureaucracy is expensive".
Post by Lux, James P
the US government will not pay for interest on borrowed money.
Don't tell the folks holding all of those US bonds!

I'm pretty sure you meant that it will not allow its subsidiaries to
borrow money separately.
Post by Lux, James P
Most capital investments have to be approved by an act of
congress, and the amortization of that investment requires special
treatment to make sure that costs are properly allocated to each
project. Leasing makes it easy. Accountants cost money too, if you do
march down the amortization process. Many, many commercial
companies have similar sorts of issues, particularly with respect
to transactions between divisions (e.g. see Regulation W for
banking), all designed to prevent "hiding profits".
Which leads to so much overhead that we end up with the $700 hammer (for
military work, which has even more of it), and at JPL, hardware prices
much above what an individual would pay to purchase the same item on the
open market. (Without benefit of a volume discount). There should
be a happy medium between using regulations to squeeze out waste/fraud,
and drowning in red tape. To me this program seems rather closer to
the latter than the former.
Post by Lux, James P
Actually, before they went to a centralized support model, they found
that the "shadow administrator" and "shadow support staff" costs were
huge, and more to the point, not accurately measureable (what you
can't measure, you can't manage).
Conversely, it may cost more to measure and manage than it saves.
Post by Lux, James P
And, while most JPL technical staff are certainly capable of doing
their own support, there's good reason for them not to
Agreed. However, there is a simple way to enforce that, deny end
users admin access to JPL supplied PCs. (Well, it might not so simple
to keep the technically adept out of their machines. )
Post by Lux, James P
Support isn't trivial here, even for office staff. For a variety of
reasons (and not unique to JPL.. Any other 5000+ employee, $1B year
business will have similar ones), we have an amazingly wide array of
various and sundry institutional applications to do things like
timecards, keeping track of inventory, document mangement, etc. I'd
venture that the desktop support staff spends more than 70% of their
time dealing with non-OS related software issues (e.g. why is my email
not getting through), and a very tiny fraction of their time responding
to hardware problems or OS issues.

Point 1, I believe you that support isn't spending much time fixing
hardware, which is why I think this contract is too expensive. Point 2,
if the core software (presumably mostly web based at this juncture) is
so problematic, it would suggest that that is the place to go for cost
savings.
Post by Lux, James P
Let us not forget IT security. This is a non-trivial matter when you
have to manage tens of thousands of desktops and comply with dozens of
pages of government regulation, NASA procedural instructions, etc.
Post by Lux, James P
The upshot is, you get a fair amount for your $130/month support
subscription. To put that in context, that's less than two hours of
engineer time.

So $4680 for three years for purchase and support of what type of
machine exactly? Does the XXX/month include the support of the
centralized business software, or just the end user's machine? It was
my impression that it was the latter, which is why I thought this manner
of supplying computers to be overly expensive. Surely network support
is covered in some sort of overhead and isn't allocated on a machine by
machine basis.
Post by Lux, James P
For what it's worth, we have a similar sort of scenario when dealing
with test equipment. Do you own it, and have in house inventory,
(with all the peculiar government contract stuff about cost
accounting), or do you have an outside vendor provide it
on lease/rent.
That's a tough one. I would assume that things like oscilloscopes and
voltmeters would be owned and kept in a pool for checkout. These have a
long service life and aren't all that expensive. For specialized and
expensive equipment, it is too complex to generalize, even before the
government purchase rules are thrown into the mix. These sorts of tools
often need expensive service contracts if purchased (to cover the
replacement of failed parts which are not generally available), and it
often comes down to 6 of one and half dozen of the other if buy or lease
is most cost effective.

It's a different problem though. Most test equipment is not a
commodity. Computers are.

Regards,

David Mathog
***@caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech
Lux, James P
2008-09-17 21:12:52 UTC
Permalink
Post by David Mathog
Translation: "bureaucracy is expensive".
You betcha.. Partly it's paranoia.. A lot of money goes into making sure that the taxpayer gets "the best deal".
On the other hand, that's not entirely paranoid. Government procurements are a target rich environment for would be thieves.
Post by David Mathog
Post by Lux, James P
the US government will not pay for interest on borrowed money.
Don't tell the folks holding all of those US bonds!
I'm pretty sure you meant that it will not allow its
subsidiaries to borrow money separately.
More technically, a government agency can't pay interest, nor can a vendor charge for "cost of money" on a cost reimbursement contract. (On Fixed Price, you can do whatever you like, except pay bribes, etc.). The ONLY way the government pays interest is those bonds. (I think..)
Post by David Mathog
to prevent
Post by Lux, James P
"hiding profits".
Which leads to so much overhead that we end up with the $700
hammer (for military work, which has even more of it), and at
JPL, hardware prices much above what an individual would pay
to purchase the same item on the open market.
Actually, I think we pay pretty much open market prices. If I "buy" a computer or peripheral, it comes from one of the big mail order places, or is at a price competitive with them, and the added "instutitional compliance costs" (which includes all the financial and acquisitions infrastructure.... I don't actually do the purchase order) is in the 10-15% range. For small purchases (that hammer), we use a government credit card and pay the consumer cash price. (of course, those self same cards feature in spectacular news stories about people making the down payment on new cars, etc.... But that's just plain old fraud.. Same as happens in any company, but it does get more publicity)


(Without
Post by David Mathog
benefit of a volume discount). There should be a happy
medium between using regulations to squeeze out waste/fraud,
and drowning in red tape. To me this program seems rather
closer to the latter than the former.
It's definitely a pendulum that swings back and forth.
Post by David Mathog
Post by Lux, James P
And, while most JPL technical staff are certainly capable of doing
their own support, there's good reason for them not to
Agreed. However, there is a simple way to enforce that, deny
end users admin access to JPL supplied PCs. (Well, it might
not so simple to keep the technically adept out of their machines. )
And, it would probably cause a riot (however, I will say that the "subscribed desktop user" is having less and less control over what's going on in their computer over time, admin privileges notwithstanding..)
Post by David Mathog
Point 1, I believe you that support isn't spending much time
fixing hardware, which is why I think this contract is too
expensive. Point 2, if the core software (presumably mostly
web based at this juncture) is so problematic, it would
suggest that that is the place to go for cost savings.
Core software is actually not web based. It's regular old MS Office, Meeting Maker, email clients (choose among Eudora, Outlook, Outlook Express, Thunderbird, and gosh, probably a dozen others), web browsers (IE,Netscape,Firefox are the leaders, but there's others), all with a variety of versions and update cycles. Then you've got Acrobat (several flavors), drawing tools (Visio, Autocad, etc.), project management (MS Project, Primavera), budgeting tools (mostly very complex excel spreadshets with pages and pages of code behind them). Then, there's various document repositories (DocuShare,Sherpa, etc.)

No question that a goodly fraction of support cost goes there. (On a positive note, the new CIO is pushing towards an enterprise architecture, which, in the long run, will reduce costs, if only because you won't have multiple parallel data sources that aren't always synced)
Post by David Mathog
Post by Lux, James P
Let us not forget IT security. This is a non-trivial matter when you
have to manage tens of thousands of desktops and comply with
dozens of pages of government regulation, NASA procedural
instructions, etc.
Post by Lux, James P
The upshot is, you get a fair amount for your $130/month support
subscription. To put that in context, that's less than two
hours of engineer time.
So $4680 for three years for purchase and support of what
type of machine exactly? Does the XXX/month include the
support of the centralized business software, or just the end
user's machine?
That supports everything, including all the applications software. It covers all skill levels of users, too. It also covers user support for institutional applications (e.g. if you have a problem getting the timecard up in the browser..)


It was my impression that it was the latter,
Post by David Mathog
which is why I thought this manner of supplying computers to
be overly expensive.
No the hardware cost is separate.. And competitively bid. Ranges from $20/month for a low end thin client to $200/month for a top-o-the-line workstation from Dell or HP, etc.

Surely network support is covered in
Post by David Mathog
some sort of overhead and isn't allocated on a machine by
machine basis.
Allocated on a machine by machine basis, because it has to be allocable to a specific project and task. The $130/mo doesn't cover "infrastructure" (e.g. fiber in the streets, router down the hall, etc.).. Those costs are also allocated piecemeal. (e.g. the phone on my desk, etc.)
Post by David Mathog
Post by Lux, James P
For what it's worth, we have a similar sort of scenario
when dealing
Post by Lux, James P
with test equipment. Do you own it, and have in house
inventory, (with
Post by Lux, James P
all the peculiar government contract stuff about cost
accounting), or
Post by Lux, James P
do you have an outside vendor provide it on lease/rent.
That's a tough one. I would assume that things like
oscilloscopes and voltmeters would be owned and kept in a
pool for checkout. These have a long service life and aren't
all that expensive.
But they need calibration every 6 months or a year, and while "functional" for decades, at some point, it's not worth servicing. There's also an intangible cost to making your engineers use clunky old gear (i.e. they're a lot more productive with a new Vector Network Analyzer than an old one, and orders of magnitude more productive than using a slotted line and a diode probe.)

For specialized and expensive equipment,
Post by David Mathog
it is too complex to generalize, even before the government
purchase rules are thrown into the mix. These sorts of tools
often need expensive service contracts if purchased (to cover
the replacement of failed parts which are not generally
available), and it often comes down to 6 of one and half
dozen of the other if buy or lease is most cost effective.
And the test equipment market IS competitive.. If vendor A thinks they can rent you the scope for $200 and Vendor B rents it for $240, Vendor A will get the job. The rental company cost structure isn't all that much different from the would be purchaser's, because you're all buying from the same vendor (e.g. Agilent).


Jim
David Mathog
2008-09-17 23:06:48 UTC
Permalink
Getting back to the original subject, what would this Cray box "look
like" when it is running windows? Does it show up as one desktop for
everything (basically an SMP machine), one desktop per blade, one per
processor(or core), or even virtualized, with more than one desktop per
core? In terms of administering the box the first of these would be by
far the easiest to deal with, since there would only be the one copy of
Windows present.

Regards,

David Mathog
***@caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech
Eric Thibodeau
2008-09-18 00:40:44 UTC
Permalink
Post by David Mathog
Getting back to the original subject, what would this Cray box "look
like" when it is running windows? Does it show up as one desktop for
everything (basically an SMP machine), one desktop per blade, one per
processor(or core), or even virtualized, with more than one desktop per
core? In terms of administering the box the first of these would be by
far the easiest to deal with, since there would only be the one copy of
Windows present.
I seriously doubt that MS is presenting the entire system as a huge SMP.
If it's the case, I'd stay away from it since it implies that either you
have to use a proprietary API to get performance (inter-core
communications à la MPI) or that the model is OpenMosix ish...which IMHO
is a nice theory, horrible practice model. My impression/technical view
of it is that the system most probably runs off a "master" board with
slave boards which boot using a network image (pretty much like NFS
roots). It is the most logical approach, again, IMHO (single point of
management and all)
Post by David Mathog
Regards,
David Mathog
Manager, Sequence Analysis Facility, Biology Division, Caltech
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
Robert G. Brown
2008-09-18 03:36:40 UTC
Permalink
Post by David Mathog
Getting back to the original subject, what would this Cray box "look
like" when it is running windows? Does it show up as one desktop for
everything (basically an SMP machine), one desktop per blade, one per
processor(or core), or even virtualized, with more than one desktop per
core? In terms of administering the box the first of these would be by
far the easiest to deal with, since there would only be the one copy of
Windows present.
I seriously doubt that MS is presenting the entire system as a huge SMP. If
it's the case, I'd stay away from it since it implies that either you have to
use a proprietary API to get performance (inter-core communications à la MPI)
or that the model is OpenMosix ish...which IMHO is a nice theory, horrible
practice model. My impression/technical view of it is that the system most
probably runs off a "master" board with slave boards which boot using a
network image (pretty much like NFS roots). It is the most logical approach,
again, IMHO (single point of management and all)
I thought it had already been said that it runs MPI, so it isn't "real
SMP" (whatever that means). I would put it a different way -- it runs N
kernels, not one N-way kernel, and it has to simulate shared memory as
there is (probably) no hardware supported NUMA or the like across the
processors. Or rather probably doesn't support anything like NUMA and
just uses message passing on top of a very normal looking
beowulf/cluster architecture. Didn't it say that in the original
article? Or was it in one of the reposts of the article?

I think the view of the universe using MS "clustering" is that all the
systems boot up a "headless windows" of some sort, and one can
access/configure the node OS's via rdesktop or some such. But I'm
guessing they have it preconfigured so that an MPI task on the "head
node" automagically distributes on the worker nodes.

rgb
Post by David Mathog
Regards,
David Mathog
Manager, Sequence Analysis Facility, Biology Division, Caltech
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
--
Robert G. Brown Phone(cell): 1-919-280-8443
Duke University Physics Dept, Box 90305
Durham, N.C. 27708-0305
Web: http://www.phy.duke.edu/~rgb
Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php
Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977
Gus Correa
2008-09-18 01:15:56 UTC
Permalink
Hello David and other Beowulf fans

Please, correct me if I am wrong.

Looking at the Cray CX1 promotional materials (brochure, movie,
press-releases),
I've got the impression that it is just a nicely packed set of blade nodes,
with GigE and IB, not very different from many clusters, except for
layout and price.
They don't say it would work as a single SMP machine, for instance, as
David speculated it might.
Apparently it is not free even from cable connections, since the back of the
node chassis look pretty standard, with GigE and IB ports, USB, etc.
However, the enclosure may have a backplane that doesn't appear on the
pictures.

Windows is the suggested OS, but they say it can run RHEL or Suse (Suse
is mentioned only on the movie).
The movie has the expected blurb about ease of use, low sys admin
requirement, etc.

Quad- and dual- core Xeon processors can be chosen from.
There are five node options: single-socket and dual-socket computing
node options,
two storage node options (4 or 8 disks), and a viz node option (with
NVidia Quadro FX4600).
The brochure says the basic power supply (for one enclosure, I presume)
is 1600W,
but you need two to be "fully populated", and there are options for
redundant PS (1+1 or 2+2).
I presume eight dual-socket quad-core compute blades would require two
active PS, right?
A lot of power for office environment.
The machine can be used as deskside (for +$520) or rackmounted.

After I configured it with eight dual-slot quad-core Xeon E5472 (3.0GHz)
compute nodes,
2GB/core RAM, IPMI, 12-port DDR IB switch (their smallest),
MS Windows installed, with one year standard 9-5 support, and onsite
installation,
the price was over $82k.
It sounds pricey to me, for an 8 node cluster.
Storage or viz node choices, 24-port IB to connect to other enclosures,
etc,
are even more expensive.

Gus Correa
--
---------------------------------------------------------------------
Gustavo J. Ponce Correa, PhD - Email: ***@ldeo.columbia.edu
Lamont-Doherty Earth Observatory - Columbia University
P.O. Box 1000 [61 Route 9W] - Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------
Post by David Mathog
Getting back to the original subject, what would this Cray box "look
like" when it is running windows? Does it show up as one desktop for
everything (basically an SMP machine), one desktop per blade, one per
processor(or core), or even virtualized, with more than one desktop per
core? In terms of administering the box the first of these would be by
far the easiest to deal with, since there would only be the one copy of
Windows present.
Regards,
David Mathog
Manager, Sequence Analysis Facility, Biology Division, Caltech
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
Robert G. Brown
2008-09-18 14:22:19 UTC
Permalink
Post by Gus Correa
After I configured it with eight dual-slot quad-core Xeon E5472 (3.0GHz)
compute nodes,
2GB/core RAM, IPMI, 12-port DDR IB switch (their smallest),
MS Windows installed, with one year standard 9-5 support, and onsite
installation,
the price was over $82k.
It sounds pricey to me, for an 8 node cluster.
Storage or viz node choices, 24-port IB to connect to other enclosures, etc,
are even more expensive.
Again, excellently well put. This is literally the bottom line. What
we are really talking about is form factor and who does what. People
usually are pretty careful with their money, at least within their range
of knowledge. When bladed systems first started coming out -- which was
many years ago at this point -- I did a bit of an on-list CBA of them
and concluded that there was a price premium of something like a factor
of 2 for them, compared to the price of an equivalent stack of
rackmounted nodes, more like 3 compared to a shelf full of tower units.
I asked "why would anyone pay that"?

The answer (given by Greg, IIRC but it was long ago and I could be
mistaken about person and maybe the answer) was:

* Cluster people with significant constraints on space, power, or AC.

* businesses that want a turnkey system, typically for HA
applications, that is compact and "easy" to support.

And that is fair enough, actually. Some places one literally has a
closet to put one's cluster in, and if one populates the closet with a
rack one has to work within the current limits on processor (core)
density per U in rackmount configuration. At the time that was
typically two, and a blade chassis could achieve something like 2-3x the
processor density per U. Space renovation can be very expensive, and
sometimes there IS no space for a room full of racks at ANY cost, and
shelved towers are even lower density as they waste a whole lot of space
per tower.

It sounds like these systems are following similar rules and economics
today. At $10K/node, it sounds like it isn't really 3x as expensive as
a rackmount cluster (with similar power) any more -- perhaps 2x as
expensive or even a bit less. If one counts it as a "64-core cluster"
it sounds a lot better than an "8-node cluster", and it also frees one
to at least consider 16 node solutions with single quad cores to reach
the same core count, that might well be cheaper (although networking
might make that not so, but where splitting up the cores provides a lot
more BW per core, so it might still be a good idea).

Still, having a 64 node cluster that fits in a few U or in a box I could
put in the corner of my desk is definitely appealing, even though the
price is for me exorbitant. A final question remains, though -- it
isn't just size (volume) -- it is infrastructure in general. How much
power does this puppy require? How much AC?

Not at all an idle question. Penguin's rackmount dual-quad nodes come
with 600 W power supplies! If an 8-core node draws a not unreasonable
400W under load, 8 nodes in a bladed chassis will ALSO draw at least 3
KW! Ouch! That's hot!

With this guestimate we have clearly exceeded the capacity of a standard
20 amp 120VAC circuit, the kind one MIGHT actually find in an office.
Even a 200W/node box is clearly beyond the capacity of a 15 amp circuit.

Big problem. My office actually doesn't have a plug or service capable
of providing 3 KW at any voltage. And it might well need 4 KW if node
draw under load is closer to 500W!

Then there is the AC problem. My office isn't too big -- maybe 60-70 sf
-- and of course it doesn't have its own thermostat or climate control
-- it is temperature controlled by a large building-scale system that
generated conditioned airflow to and between all the rooms on my hall.
It stays perfectly comfortable with a variable load of perhaps 400-700 W
of power produced within its walls -- that would be me (100W), lights
(100W), a desktop (150W), a laptop (50W), and a bit of this and that,
including a few students from time to time at 100W each and THEIR
laptops as well, come to think of it. So it could have as much as a KW
peak heat load, but it's a big building, there is some "ballast" and
averaging from neighboring offices, and it doesn't really get hot even
when we party in their.

Now throw that extra 3 KW in. Suddenly the MINIMUM load is over 3x the
PEAK load now. My office has no thermostat -- I just have to live with
what the building delivers. Room temperature soars. rgb has to start
going to work in shorts and sandals, no shirt, and installs a keg
refrigerator to hold the beer essential to life that must be piped
directly to mouth to replace the sweat and dull the pain of working
under such extremely hot conditions. Only for a short while, of course
-- cluster dies a horrible death, as it isn't designed to run at 110F.

And then along comes winter, and Duke, foolishly assuming that it is
cold outside and people need HEAT and not COLD delivered to their
offices, shuts off the building chiller and turns on the steam.
Obviously I wouldn't need the steam, but EVEN IN THE WINTERTIME I won't
be able to get rid of 3500-4500 watts. I'd need a TWO-ton (well, I
might get away with a 1.5 ton) AC unit of my very own to remove this
heat and dump it -- somewhere -- and I'd have to control its thermostat.

Fine. One can, actually buy 1 ton to 1.5 ton AC units that sit over in
a corner and vent heat into e.g. the space inside a drop ceiling (where
eventually building AC/climate control sucks it up and re-air-conditions
it away). They draw a few hundred watts of their own, need a place to
dump water from condensation (or a human has to hand-empty their
condensation pan), and of course they are loud and annoying, but they
work, when somebody doesn't kick the drain tube out so that they leak
all over the floor or the ceiling space gets so hot that hot air back up
into the room anyway.

So, to run one of these "office clusters" in the REAL world, it sounds
like I'd probably need the following:

a) An additional 4 KW-capable circuit in my office. Concrete walls,
I'm guessing fully subscribed service, let's be optimistic and add just
$1K to get it. That leaves me my existing 20 A service to run the AC
on; tight but doable -- or of course I can get a couple of high-power
circuits while I'm at it, one for the system and one for its AC.

b) A 1.5 ton AC as described. There goes 1 m^2 of floor space, hmmm,
where the hell am I going to get THAT? A filing cabinet must go, or
students will just have to sit on it. The unit costs perhaps $3-5K,
depending on new or used and condition etc, and it still needs a bit of
installation beyond just buying it from a catalog. Oops, and my office
doesn't, actually have drop ceilings. Need to vent it outdoors, need
hole in wall for vent, need at least $1K to pay the nice man to put it
there and make it all pretty. Double oops, that needs outside wall,
guess I have to lose a bookshelf intead of a filing cabinet, stack books
to ceiling on top of filing cabinets that remain.

c) A drain. Another hole. No bathroom nearby, I guess we'll have to
just drill a different wall and dump. Hmmm, I wonder if local codes --
or Duke -- will let me do that? A perpetual drip into an interior
"chimney" airvent next to the stairwell has nowhere to go. One out the
other window, where will that water go? Must check, but either way
another $500 for sure (and I'll have to move completely out of the
office while they chop holes and generate all that dust, brick and
concrete walls, very difficult to drill).

So, after budgeting around $90K -- or even more -- for actually putting
my cute little 64 core system, after moving out of my office while power
and AC are retrofitted, after moving back into my office and clearing
room on top of a filing cabinet or on a corner of my desk for the blade
chassis, I move back in, crank the AC up to max (that's what it will run
on nearly all the time, max) and power up my little sucker. Its cooling
fans -- capable of moving that 3+ KW out of its small volume -- kick on
and add to the soothing, um, "purr" of the AC, and I realize that I'm
going to be living at 40 dB or more of steady-state background noise
forever, maybe even 50 by the time my other hardware kicks in. Students
have to speak loudly to overcome the noise, I take to wearing
noise-cancellation headphones all the time even when I'm not listening
to music in my room just to protect my hearing. But I've got a hell of
a cluster, and it is in my office, and I snuck that keg refrigerator
into the budget by arguing that the cluster was "liquid cooled"!
Coolio!

OR, since we already HAVE a lovely server/cluster room downstairs, with
a large AC unit built right in that keeps the room COLD, and lots of
power-poles just waiting to deliver power and trivially rewireable to
deliver e.g. 209 VAC at 30A and no humans who have to listen to the
60-80 dB background roar except for brief intervals doing service, I
COULD just buy a stack of 1U sufficient to give me 64 cores with
2GB/core and high speed networking, in the cheapest (per core) rackmount
form factor. I don't even have to buy rack space, as we have that in
abundance. Pricing out reasonably comparable 1U nodes from e.g. Penguin
it looks like they would cost "around" $6000 with the usual service etc
-- around $750/core instead of $1100/core (including the infrastructure
mods). The cluster costs me a bit over 1/2 as much -- $50K, say --
although I'm not actually pricing it out and if I shopped harder it
might be even less, or a bit more if I forgot something in my estimate.

Now, I don't have to listen to it. I don't have to rewire my room and
give up a table, desk space, etc. I have network access in my office
(obviously) -- multiple ports, actually. If I feel really ambitious I
spend a little of the money I save arranging to pipe one of the ports
directly into my cluster down there so I can make my desktop a kind of
"head node" for the cluster, although in practice I've never done that,
why bother? GigE access on our standard LAN seems adequate, somehow.

With the money I save I buy, lessee, at least 48 more cores! Wow, I get
my computations done in a lot less time! Wow, with with 112 3 GHz cores
total, I actually have capacity to spare, and can even loan cores out to
my buddies who are CPU starved, which inspires the University to pay the
nontrivial cost of providing power and AC to all of these CPUs! The
cost savings of MY TIME -- well, it's hard to compute, but simply taking
the salary I would need to be paid while all that work takes well over
half again as long to complete and feeding even the savings on my meager
salary back into still more nodes, the break-even point on the CBA
exercise would probably leave me with WELL OVER 2x the 64 core capacity
of the "cute" desktop cluster model.

This little exercise in the realities of infrastructure planning exposes
the fallacy of the "desktop cluster" in MOST office environments,
including research miniclusters in a lot of University settings. There
exist spaces -- perhaps big labs, with their own dedicated climate
control and lots of power -- where one could indeed plug right in and
run, but your typical office or cubicle is not one of them. Those same
spaces have room for racks, of course, if they have room for a high
density blade chassis.

If you already have, or commit to building, an infrastructure space with
rack room, real AC, real power, you have to look SERIOUSLY at whether
you want to pay the price premium for small-form factor solutions. But
that premium is a lot smaller than it was eight or so years ago, and
there ARE places with that proverbial broom closet or office that is the
ONLY place one can put a cluster. For them, even with the relatively
minor renovations needed to handle 3-4 KW in a small space, it might
well be worth it.

rgb
--
Robert G. Brown Phone(cell): 1-919-280-8443
Duke University Physics Dept, Box 90305
Durham, N.C. 27708-0305
Web: http://www.phy.duke.edu/~rgb
Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php
Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977
Mark Hahn
2008-09-18 15:56:37 UTC
Permalink
Post by Robert G. Brown
* Cluster people with significant constraints on space, power, or AC.
just space, really. blade systems used to be almost unique in offering
high-efficiency power solutions, but I think most or all that's become
available in the commodity market now. (that is, 80-90% psu's in normal
1U servers).

and remember, less space and approximately the same power means higher
heat-density. I've never seen a lot of fully populated blade enclosures
in one spot (which is kinda the point), though it should be doable with
rack-back heat exchangers.

actually, are there any blade systems that skip air-cooling entirely?
that would actually make sense - if you're going to go for bespoke power
because of potentially greater efficiency, bespoke cooling makes sense
for the same reason.
Post by Robert G. Brown
* businesses that want a turnkey system, typically for HA
applications, that is compact and "easy" to support.
that part never made sense to me. I'm skeptical that the management
interface for blade systems is better than plain old IPMI. prettier,
perhaps.
Post by Robert G. Brown
And that is fair enough, actually. Some places one literally has a
closet to put one's cluster in, and if one populates the closet with a
a closet which just happens to have a huge jet of cold air going through it...

http://www.cray.com/Products/CX1/Product/Specifications.aspx
claims 1600W, 92% efficient. their pages don't give much info on the
engineering of the blades, though. given that you have to add ipmi
as an option card, it looks pretty close to commodity parts to me.
Gus Correa
2008-09-18 17:56:01 UTC
Permalink
Hi Mark and list
Post by Mark Hahn
http://www.cray.com/Products/CX1/Product/Specifications.aspx
claims 1600W, 92% efficient. their pages don't give much info on the
engineering of the blades, though. given that you have to add ipmi
as an option card, it looks pretty close to commodity parts to me.
Note that for the fully populated enclosure the power supplies are dual,
i.e.
2* 1600W = 3200W.
See my posting:
http://www.beowulf.org/archive/2008-September/023176.html

3200W makes sense if you have 64 Xeon cores inside,
and matches also RGB's precise guesstimate of 3kW.
(It is always prudent to have any computer marketing blurb distilled and
deconstructed
by a bona fide physicist.)

Indeed, the Cray CX1 promotional materials don't say much about which
value Cray added to the hardware, and not to the software either, for
that matter.
It looks like just as a nicely packed set of blade servers,
of which there are so many others out there.

The article/interview on HPC Wire has several interesting points raised by
Cray's senior VP of sales and marketing:

For instance:

** Number 1) "It's designed to run on standard 110 volt office power,
and doesn't require additional cooling. It can sit under your desk,
right where you work."

Questions:
110V, but how many Amperes?
Does a typical office electrical installation support this? ( I can't
avoid a grin :) )

As for the cooling requirements, please see RGB's humorous but very
sharp and to the point posting.

**

** Number 2) "Second, we've tried to lower the bar in terms of the
talent required to deploy one of these at a customer site. In terms of
infrastructure, the CX1 will just drop into a standard office
environment. The operating system and tools will be very familiar to
Linux or Windows users,"

Questions (same RGB asked):
What environmental conditions should such an office have?
1.5ton A/C?
4kW capable wiring?
Beer keg refrigerator?
How many air vent and drip holes on the walls, ceiling and windows?

Somebody already commented about:

"we've tried to lower the bar in terms of the talent required to deploy
one of these at a customer site"

However, my feeling is that teaching 101 courses in Unix/Linux proficiency
(directory tree, command redirection, etc), Unix programming environment
(make, etc),
and Unix tools (vi or emacs, sed, awk, basic shell scripting, etc),
for science and engineering students would increase the "talent required"
at a much lower cost and with much higher benefits than buying Windows
deskside supercomputers.
Who would need a Windows based HPC then?
To entice the freshmen students, RGB could give a few cool special
lectures about Turing machines,
cellular automata, etc, on these 101 classes.
This would pay off much better than teaching C and C++ (say, with
Visual Studio) to freshmen,
would give them a background to use Unix/Linux machines effectively,
and would prepare them to go beyond Matlab and Windows.
Programming languages could be 201 or higher level courses.

**

*** Number 3) "Finally, the CX1 starts at just $25,000, with fully
configured systems reaching the $80,000 range. The system is very
affordable in terms of both the initial capital investment and the
lowered total cost of ownership customers will see from ease of
management and standard office power and cooling requirements."

Questions:
How many blades and cores come in the $25k configuration? (I will
answer: One blade, just like a
dual-socked quad-core workstation that you can buy for $5k or less.)
How "affordable" an investment you need to make install it in your
office? (See RGB's posting for the answers.)

**

**** Number 4) "We believe that there are many workstation users today
who are used to working in a Windows environment and find the thought of
moving to a more powerful platform like an HPC cluster and the challenge
of learning a new operating system daunting. By offering an operating
system that they are familiar with, we believe the barriers to adoption
are significantly lowered. "

Comments
The system seems to be a regular cluster, nicely packed, and perhaps
with an easy setup procedure.
The main attraction is that it can run Windows, for those who are afraid
of Linux.
Other attractions are the nice looking enclosure and the brand name,
that make it an object of desire.

***

The problem with unsubstantiated statements like these on the HPC Wire
interview
is that they catch the attention of decision makers (Deans, department
heads,
as someone mentioned here), and you may have a hard time to distill and
deconstruct them.
Unless the decision maker has a background or a very good guts felling
for computers and
the underlying physics/engineering, the shiny brochures, the movies,
and the big brand names can really make a dent.

I had to write a long explanation, going down to details very similar to
those RGB raised here
(but not with the same sharp humor), to justify buying a cluster as
opposed to a
"turnkey" solution akin to this "deskside supercomputer" just two weeks
ago.
I went through the same arguments that RGB used here:
environmental issues (power, A/C, floor space), TCO,
sys admin and maintenance, pros and cons of COTS vs. proprietary HW and
SW, etc.
I won, but now I'll probably have to "dejavu it all over again",
when people here learn about the CX1.

If my boss was not knowledgeable in Physics and computing,
I would have to invite RGB to come here to give my boss a briefing.
And to recommend buying a beer keg refrigerator along with a cluster, of
course.

Gus Correa
--
---------------------------------------------------------------------
Gustavo J. Ponce Correa, PhD - Email: ***@ldeo.columbia.edu
Lamont-Doherty Earth Observatory - Columbia University
P.O. Box 1000 [61 Route 9W] - Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------
Post by Mark Hahn
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
Mark Hahn
2008-09-18 18:21:57 UTC
Permalink
Note that for the fully populated enclosure the power supplies are dual, i.e.
2* 1600W = 3200W.
I see that now. question for electrical-code enthusiasts on the list:
isn't a normal "15 amp" office circuit actually supposed to only be
used up to 12 amps? that would mean 12*120=1440; in other words, each
1600W PS needs a special circuit. also, isn't it pretty common for
an office duplex receptacle to be fed by a single (15A) circuit?

I guess what really surprises me is that Cray doesn't mention ScaleMP,
which would seem to suit this box ideally. if someone wants Windows,
I'd very much expect them to want the convenience of ignoring the fact
that memory is distributed...
Michael H. Frese
2008-09-18 18:51:07 UTC
Permalink
My office clusters are plugged into 25 A circuits that are relatively
common in office buildings. That still doesn't quite get you 3
kW. But any company spending $80k for a computer will spend a
kilobuck or so more to upgrade the wiring and panel.
Post by Mark Hahn
Note that for the fully populated enclosure the power supplies are dual, i.e.
2* 1600W = 3200W.
isn't a normal "15 amp" office circuit actually supposed to only be
used up to 12 amps? that would mean 12*120=1440; in other words,
each 1600W PS needs a special circuit. also, isn't it pretty common
for an office duplex receptacle to be fed by a single (15A) circuit?
I guess what really surprises me is that Cray doesn't mention ScaleMP,
which would seem to suit this box ideally. if someone wants Windows,
I'd very much expect them to want the convenience of ignoring the fact
that memory is distributed...
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
Lux, James P
2008-09-18 20:20:34 UTC
Permalink
Post by Lux, James P
-----Original Message-----
Sent: Thursday, September 18, 2008 11:22 AM
To: Gus Correa
Cc: Beowulf
Subject: Re: [Beowulf] Re: MS Cray
Post by Gus Correa
Note that for the fully populated enclosure the power
supplies are dual, i.e.
Post by Gus Correa
2* 1600W = 3200W.
isn't a normal "15 amp" office circuit actually supposed to
only be used up to 12 amps? that would mean 12*120=1440; in
other words, each 1600W PS needs a special circuit. also,
isn't it pretty common for an office duplex receptacle to be
fed by a single (15A) circuit?
Don't confuse a 15A receptacle with a 15A circuit. You can have multiple 15A receptacles on a 20A circuit.

A 20 A receptacle is also not uncommon (the one with the T shaped slot).. The 20A plug has one pin/blade at 90 degrees relative to the other.

20A circuits are pretty common.

And, in a lot of offices, multiple circuits wind up in each office (modular wiring systems, for instance, will carry all three phases of a 208/120Y system), so with two power cords you could do the 3kW.

And, hey, if you're spending 90K on the box,you can spend 1K on the electrician to install a 30A circuit.


Jim
k***@neuralbs.com
2008-09-18 20:36:27 UTC
Permalink
Post by Mark Hahn
Note that for the fully populated enclosure the power supplies are dual, i.e.
2* 1600W = 3200W.
isn't a normal "15 amp" office circuit actually supposed to only be
used up to 12 amps? that would mean 12*120=1440; in other words, each
1600W PS needs a special circuit. also, isn't it pretty common for
an office duplex receptacle to be fed by a single (15A) circuit?
Here is the _practical_ answer: YES. And here is why:

Loading Image...
Post by Mark Hahn
I guess what really surprises me is that Cray doesn't mention ScaleMP,
which would seem to suit this box ideally. if someone wants Windows,
I'd very much expect them to want the convenience of ignoring the fact
that memory is distributed...
Yes, that is what I was also asking in a spinoff thread and doubted that
it would have been the case.

IMHO, anyone is better off buying a Tyan-type SMP such as the VX50 we have
here. For about 25k$ CAD, we got 8x2core Opteron with 32Gigs of
ram...that's over a year ago and it's been running flawlessly since I
built it (except for dead ram due to power fluctuation/outage)

Eric
Shai Fultheim (Shai@ScaleMP.com)
2008-10-19 03:37:45 UTC
Permalink
Mark,

Thanks for the kind comment. ScaleMP does not (yet) support MS-Windows OS.


--Shai


-----Original Message-----
From: beowulf-***@beowulf.org [mailto:beowulf-***@beowulf.org] On Behalf Of Mark Hahn
Sent: Thursday, September 18, 2008 11:22
To: Gus Correa
Cc: Beowulf
Subject: Re: [Beowulf] Re: MS Cray
Note that for the fully populated enclosure the power supplies are dual, i.e.
2* 1600W = 3200W.
I see that now. question for electrical-code enthusiasts on the list:
isn't a normal "15 amp" office circuit actually supposed to only be
used up to 12 amps? that would mean 12*120=1440; in other words, each
1600W PS needs a special circuit. also, isn't it pretty common for
an office duplex receptacle to be fed by a single (15A) circuit?

I guess what really surprises me is that Cray doesn't mention ScaleMP,
which would seem to suit this box ideally. if someone wants Windows,
I'd very much expect them to want the convenience of ignoring the fact
that memory is distributed...
_______________________________________________
Beowulf mailing list, ***@beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
Lux, James P
2008-09-18 18:39:08 UTC
Permalink
Post by Gus Correa
Other attractions are the nice looking enclosure and the
brand name, that make it an object of desire.
Come on now.. Haven't you always wanted a *Cray* of your very own?

Think of it like a BMW 1-series or Baby Benz

Jim
Joshua Baker-LePain
2008-09-18 18:49:22 UTC
Permalink
Post by Lux, James P
Post by Gus Correa
Other attractions are the nice looking enclosure and the
brand name, that make it an object of desire.
Come on now.. Haven't you always wanted a *Cray* of your very own?
Think of it like a BMW 1-series or Baby Benz
But what good is your own Cray if it doesn't have a waterfall?
--
Joshua Baker-LePain
QB3 Shared Cluster Sysadmin
UCSF
Mark Hahn
2008-09-18 18:50:36 UTC
Permalink
Post by Lux, James P
Post by Gus Correa
Other attractions are the nice looking enclosure and the
brand name, that make it an object of desire.
Come on now.. Haven't you always wanted a *Cray* of your very own?
Think of it like a BMW 1-series or Baby Benz
or an Apple. after all, Apple hardware is utterly mundane,
except for the packaging and bundling
(and the moist, warm comfort of being in the cult.)

the difference is that Apple's halo effect is everywhere you look.
does Cray have a non-old-fart halo?
Gus Correa
2008-09-18 22:04:33 UTC
Permalink
Hello James and list
Post by Lux, James P
Post by Gus Correa
Other attractions are the nice looking enclosure and the
brand name, that make it an object of desire.
Come on now.. Haven't you always wanted a *Cray* of your very own?
Nope.
I would be happy to use them at national facilities, TeraGrid, etc.
Real Crays belong there.
They make sense there.

Besides, other than the logo, the CX1 doesn't sound as a Cray.
Where is the the innovation in architecture design, the push for new
levels of performance?
I wonder what Seymour Cray would say of it.
Post by Lux, James P
Think of it like a BMW 1-series or Baby Benz
... Ferraris, Lamborghinis ... (yawn)
I have other aesthetic interests.

I drive a rusty 93 Honda Accord, which makes about 30 miles per gallon.
Who can ask for anything more?
Hmmm ... a new radio, maybe, to listen to WBGO 88.3 FM Jazz Newark with
less hissing.

I can ask for something less, though.
If I lived in NY City, with subway, buses, etc, I wouldn't have a car,
it would be just an unnecessary annoyance.
Not to mention Hubbert's peak and global warming.

Gus Correa
--
---------------------------------------------------------------------
Gustavo J. Ponce Correa, PhD - Email: ***@ldeo.columbia.edu
Lamont-Doherty Earth Observatory - Columbia University
P.O. Box 1000 [61 Route 9W] - Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------
Robert G. Brown
2008-09-19 11:13:58 UTC
Permalink
"we've tried to lower the bar in terms of the talent required to deploy one
of these at a customer site"
But this is always the point of a turnkey vs home-engineered system. Do
it yourself is cheaper, but you have to do it yourself. This means you
have to be CAPABLE of doing it yourself.

To my direct experience, the humans on this planet self-partition into
two groups. The partitioning isn't strictly on talent or intelligence,
although both can be factors, but on something else, an element of the
human psyche. One group can do "it" themselves and, truth be told,
PREFERS to do it themselves (for nearly any value of it). They may not
have the talent to do it when they start, but they do when they are
done within the natural limits of their God-given intelligence,
strength, wisdom, and number of hit points.

The other, simply put, does not. They do it themselves in the natural
regime of it imposed on them by a cruel nature that won't feed them or
provide them with shelter unless they do it, and bitterly resent both it
and any other it they are ever forced, kicking and screaming, to attempt
and half-assedly master. They are ever willing to let other people do
it on their behalf, and will often pay large sums of money not to have
to do it.

This partitioning extends through life from young to old, across
national and cultural boundaries, across social strata from top to
bottom (in spite of the fact that one would expect doing it to provide
social or economic benefits -- and liking to do it does -- there are
plenty of blue-collar workers who passionately try to do it within their
spectrum of talents and opportunities just as there are plenty of
lazy-ass white collar workers who do it only if it involves oddly shaped
sticks and a small white ball or telling other people to do it (the one
it people in the latter category often feel uniquely qualified to do at
all levels and values of it).

So (to go back on topic after that deep philosophical observation to a
list composed almost entirely of people who love doing it:-) lowering
the bar actually doesn't really help. Anyone actually capable of doing
it (where "it" = "program a serious parallel application involving MPI
and a pile of computers") doesn't need the bar lowered. They might go
turnkey to save time, especially when they are lavishly funded so that
they aren't forced to do it to save money to be able to afford larger
resources and actually finish faster. They might go turnkey because
they are already doing it for values of it such as "doing immense
amounts of laboratory research on genetics" so that they simply don't
have the time to mess with it -- where "it" in the latter case almost
certainly includes using a cluster to develop applications, so that
they'd also just purchase/install free turnkey software to do the
required genomics.

Nearly everyone else would RATHER do it because it is fun, because it is
satisfying, because they can get 40-60% more hardware capacity with the
money they save, because doing it themselves they learn far more in the
process and end up actually able to use the resource that they've built
at its full capacity instead of just knowing it as a sort of black box
front ended by a Visual screen that obscures all of the all-important
detail that might have one day taught them what the hell they're doing
with it so that they could do it well.

I'm not making it up. The lesson that Cray et. al. (and now Microsoft
too, jumping on the bandwagon so to speak) have apparently still failed
to learn is that the REASON that fifteen years ago nearly all the
supercomputers in the world were big iron fronted by expensive,
proprietary operating systems and custom and highly non-portable
compilers and programming tools (often purchased for millions of dollars
from Cray) where today these have all but disappeared from the world is
that the people that actually use HPC are overwhelmingly in the category
that can do it, that likes doing it, that know that doing it themselves
gives them a degree of ownership and control and knowledge that is
utterly inaccessible to those who shy away from doing it and want it
done for them, especially if they can spend other people's money to get
it without mental engagement on their part. The latter people rarely
succeed in doing it. Darwin slowly but surely acts against them in the
highly competitive research environment.

The end result is that "successful" turnkey vendors in HPC (as opposed
to HA or "server farms" aren't black-box providers, they are integrators
and educators like Joe. Joe (correct me if I'm wrong, Joe) doesn't JUST
engineer a system to meet some customer's requirements, plug it in, and
leave -- he might engineer it and build it, but he builds it WITH the
client, teaching the client what it's all about, working with the client
to the point where the client in the end has full ownership and has
learned what they need to learn to use the tool and perhaps take care of
it. Scyld/Penguin provides much the same sort of thing. Bottom line:
you don't do HPC if you aren't capable and naturally inclined to do it
yourself, and the top 500 list has over fifteen years come to
overwhelmingly -- overwhelmingly -- reflect that simple fact. This is a
true dynamic phase transition and not particularly likely to suddenly
revert to the earlier phase just because Cray and Microsoft try to get
the research world to regress to a state where customers pay THEM most
of the money they'd otherwise spend getting more processors. If they
were likely to do that now, they never would have stopped then instead
of running like bunnies away from Crays and towards commodity clusters.
However, my feeling is that teaching 101 courses in Unix/Linux proficiency
(directory tree, command redirection, etc), Unix programming environment
(make, etc),
and Unix tools (vi or emacs, sed, awk, basic shell scripting, etc),
for science and engineering students would increase the "talent required"
at a much lower cost and with much higher benefits than buying Windows
deskside supercomputers.
Who would need a Windows based HPC then?
It's far past that. Who in the world is CAPABLE of programming a
parallel HPC who doesn't already know all of these things? Does
Microsoft seriously think that there exist fifty competent programmers
in the world that can write brilliant HPC-class programs using only
Microsoft compilers and tools but who have no idea what Unix is and how
it works and how to program in a Unixoid programming environment? Not
even their OWN programmers are that ignorant -- or rather, where they
are that ignorant they give us (gulp) >>Vista<< as an example of
superbly optimized, unbloated high performance code. I can see it now:

"Trust in the people that brought you Vista to provide you with high
performance tools that will produce the fastest code that will run on
the leanest resources."

Right. Like the 4 GB needed just to BOOT THE OPERATING SYSTEM of their
latest "desktop" version of Windows so that it doesn't run like a pig.
To entice the freshmen students, RGB could give a few cool special lectures
about Turing machines,
cellular automata, etc, on these 101 classes.
This would pay off much better than teaching C and C++ (say, with Visual
Studio) to freshmen,
would give them a background to use Unix/Linux machines effectively,
and would prepare them to go beyond Matlab and Windows.
Programming languages could be 201 or higher level courses.
I don't know offhand of any University that teaches Visual Studio to
freshmen as a means to learn C or C++, period. Damn few schools teach
C++ to freshmen, and nobody nowhere teaches C. Duke certainly doesn't,
and Duke and Microsoft are practically family. Community colleges might
teach it, and there might be courses at some level that teach it, but
generally speaking computer science departments -- once they get past
the intro courses in java -- use and teach Unixoid operating systems
(overwhelmingly Linux, but still quite a lot of e.g. Solaris) and
programming tools such as C, C++ (and rarely Fortran), often at the same
time.

After all, what can you learn from a closed source broken piece of, um,
"black box" like Windows (any flavor, any time)? How to fix a broken
registry? And what operating system, exactly, is written entirely in
C++ using a visual toolset? Does any professor WANT their students to
learn how IT "works", even if they could know themselves well enough to
teach it?

The difficulty of teaching or learning Unix/C programming tools and
methods is greatly exagerrated. I already teach this -- I'm teaching it
to one student this semester who has to learn Linux and C just to work
on a neural network project; I taught a student C from scratch last
semester (and the student is this semester taking an advanced course in
operating systems and programming and writing me lavish thank you notes
as he is blowing away the course, way ahead of his classmates).

I just loan them (and have them buy) a handful of books like Kernighan
and Pike and Glass and Ables (the latter a much more modern and useful
version of the venerable, nay classic, former). I teach them to use
jove: "Hey, bring up an xterm and run 'teachjove'. Check back with me
in a couple of days." I give them or show them a few free online or PDF
C books plus teach them about the man pages. I show them how to run a
Makefile from inside jove (Ctrl-X Ctrl-E, fingers stay right on those
keys where they do all of their work). I give them a C program template
(available on my website) to make it easy to get started with a
functioning ready-to-build example. I have them write three or four
very simple programs STARTING from that template to kind of get their
feet wet. I show them how to use SVN and threaten them with the hell of
utter failure if they don't use it and somehow lose their work by
overwriting their non-backed up sources. I deconstruct their programs,
whack them upside the head with a manual whenever the ratio of comment
lines to code drops much below 1:1, tell them that unindented code is
the invention of the Devil intended to lead poor lost souls into
purgatory, that good variable names and function call names aren't too
long, aren't too short, and DontUseSillyCaPS unless they mean something.

I teach them something that Microsoft-based tool users never really
figure out (witness the earlier posting from the Microsoft person) --
there is this thing called "ASCII" which compilers like, which produces
perfectly readable straight text, and which is PORTABLE across viewing
and editing environments, proprietary and non-proprietary. Good
programmers consequently nearly invariably use a straight ASCII text
editor, and eschew the use of the 8th bit and/or multibyte characters
unless for them straight text happens to be in Chinese or Arabic. Email
messages from good programmers to lists like this rarely contain those
funny looking @?isms that basically signal to the world "Hey, I wrote
this in a Microsoft mail program that uses an extended character set
even for everyday English text characters that (of course) any Microsoft
mail program will correctly render but that will (of course) look funny
on your non-Microsoft screen, which proves that you should use Microsoft
mail programs for all purposes so you can read my mail".

I teach them that Unix tools can be used to work dark magic, but that
there is a price to be paid in sweat and chickens slaughtered with
black-handled knives before you can decide that your code needs to be
completely reorganized into subdirectories and that as part of this
reorganization every program module needs a certain code fragment to be
changed to a different code fragment and you accomplish this with three
lines from the shell in around a minute or two of actual effort. They
don't call master programmers "wizards" for nothing, but knowledge is
power, and power is never cheap.

These students are obviously in the self-sorted "I can do it myself for
any value of it" category, but the point is that all programmers ARE in
this category unless they chose their profession to make money, as the
value of "it" that they are reluctantly forced to choose in order to
stay fed and clothed. Such "programmers" as often as not become
marketing droids or pointy-haired bosses because they do not know the
Tao of Programming and rarely become more than half-competent hacks.
Only programmers who have learned to follow the Tao and slaughtered many
chickens effortlessly write code that compiles flawlessly and runs like
a category five hurricane over an ocean of processors. And they don't
do it on Windows.
*** Number 3) "Finally, the CX1 starts at just $25,000, with fully configured
systems reaching the $80,000 range. The system is very affordable in terms of
both the initial capital investment and the lowered total cost of ownership
customers will see from ease of management and standard office power and
cooling requirements."
One blade, just like a
dual-socked quad-core workstation that you can buy for $5k or less.)
How "affordable" an investment you need to make install it in your office?
(See RGB's posting for the answers.)
Just to end the speculation (however fun:-), here are a few facts from
its spec sheet. The box is 7U in height. It takes >>4<< 1600 watt
power supplies (2+2) to run the entire system in "fully populated, fully
redundant" mode. Two power supplies only let you run "half populated,
fully redundant mode" (1+1) or of course fully populated, not redundant
(2+0). One power supply can only support four blades, obviouly not
redundant, where the blades draw (as I learned from a different source)
roughly 375W each, presumably peak/loaded but the only object I'd really
trust here is a Kill-a-Watt -- well, 2 Kill-a-Watts -- plugged in
between the fully populated CX1 and the wall and running at full load
(maybe doing a good benchmark suite in parallel, so one can drive all
the CPUs to draw peak power synchronously).

So technically, the CX1 "can" run on an office power supply deskside, as
long as you only populate it with <=3 blades (15 A) or <=4 blades (20A)
and nothing else to speak of is on the circuit.

The box is a 7U pedestal: 13" by 18" by 36" (!). In other words, this
sucker is a yard long! Note that they only present you with tasteful
photos of the FRONT of the enclosure, where perspective hides this last
distasteful fact. They ONLY present this view -- check out their
website or brochure: you only see either the front panel or a picture
that at first makes it look like the entire box is the size of a
shoebox, would fit nicely on the corner of your desk.

Not.

It weighs 140 pounds fully populated, 63 pounds empty. This makes it a
bit more than 1/2 the width of a standard rack, and about as deep.

Two observations: If you have the room for this puppy "under your desk"
you have a bigger desk than I've got, and like to have very, very warm
feet. Just perfect for those supercomputer users in Iceland and Alaska,
and I'll bet Antarctic research stations are clamoring to get them and
plan to run heavy applications all the time with one on each side of
their desks.

Also, one can easily find and order portable/deskside racks in
sizes ranging from 8U to 24U, with wheels and/or shockmounts. A
top-dollar, shipping grade shockmounted 12U travel rack looks like it
costs around $1400 delivered. Into this one can put as many e.g.
Penguin dual-quads in standard 1U ff as you wish, at ~$5000 each in the
comparison configuration (3 GHz cores, 2 GB RAM/CPU). To these you can
add the network of your choice -- gigabit ethernet is "free", of course
-- inside an expected $2K budget for rack, cabling, switch, KVM, and
for a bit more UPS or other gingerbread. Faster networks obviously add
cost at market price. Eight additional inches of width and six+ in
height let you install 88 cores instead of 64 and use true commodity
parts instead of being locked forever to Cray if you want to upgrade or
change vendors.

Or, the volume occupied is very, very close to the space that four
minitowers would occupy, which is all one can actually plug into even a
lavishly wired office without renovation anyway. In this case one
spends a straight up ~$4-4.5K per box and you're done (plus network) and
once you go down this road, you can start thinking about a pile of
minitowers with just one quad each etc. I admit you don't get that
active noise cancellation, but noise cancellation headphones are $100
and play music too.
**** Number 4) "We believe that there are many workstation users today who
are used to working in a Windows environment and find the thought of moving
to a more powerful platform like an HPC cluster and the challenge of learning
a new operating system daunting. By offering an operating system that they
are familiar with, we believe the barriers to adoption are significantly
lowered. "
And I believe that they are possibly right for a fairly small value of
"many", especially if one emphasizes the term "user". This is not a
coding or development platform -- it is a platform they want to sell to
people to run canned, commercial code (and that is a sub-market right
there -- several of the blades they sell are obviously HA server blades,
not HPC). The only people likely to buy it for HPC development are
people seeking to develop canned, commercial HPC code to sell those
users, who will then use the programs to do what they are told to do
with them by bosses who themselves don't understand what the systems are
and how they work, only that they are told -- by a marketing droid --
that using the system will permit them to get their work done faster
because, well, look at all the cores! And Cray MEANS supercomputer,
doesn't it?

A true rendering of the release headline might just as well be "Cray
reinvents the SP2" -- five or ten years after the SP2 was rendered
obsolete. A commodity SP2, to be sure. We've come at least that far.

Beyond that, one can do a head to head comparison of this bladed system
and other folks' bladed or other solutions, and I think that most of
what you are getting is that ever so impressive CRAY on the front --
which has a certain cachet, does it not?;-) -- plus something that NO
DOUBT is well engineered, solid hardware, sold for a premium price, that
takes up less room than a rack of the same capacity (but well within a
factor of two in VOLUME and with at least one troublesome yard-long
dimension).
Comments
The system seems to be a regular cluster, nicely packed, and perhaps with an
easy setup procedure.
The main attraction is that it can run Windows, for those who are afraid of
Linux.
Other attractions are the nice looking enclosure and the brand name, that
make it an object of desire.
I desire it, I desire it. Hell, I'd make room for it in my life and
sweat for it just to run it. I just can't possibly PAY for it, and if I
could I still wouldn't because I could get more for my money elsewise.

Now if Cray is listening in and wants to just GIVE me one (fully
populated, please -- I did install two 20A circuits just for my personal
supercomputer setup in my attic and while I'm a BIT light on AC up there
for 3.2 KW, I can probably scare up enough OPM to do an upgrade) hey,
I'll do random number generator testing and other Good Works on it and
tell everybody on this list what a great system it is! I admit it -- at
heart I'm just a whore, easily bought:-).

Oh, but not even for free would I accept one preinstalled with Windows
unless I were permitted to just pop Fedora onto it first thing. I may
be cheaply bought but I'm not stupid, and Windows would cost me way more
than the value of the hardware in my time, on this just as it has every
other system I've ever installed or managed with Windows on it. Windows
management is expensive as all hell, whether you do it yourself or pay
others to do it for you. One of the all-time great success stories of
marketing is that Microsoft has managed to hide or obscure this fact
from a truly astounding number of people for well over a decade.
The problem with unsubstantiated statements like these on the HPC Wire
interview
is that they catch the attention of decision makers (Deans, department heads,
as someone mentioned here), and you may have a hard time to distill and
deconstruct them.
Unless the decision maker has a background or a very good guts felling for
computers and
the underlying physics/engineering, the shiny brochures, the movies,
and the big brand names can really make a dent.
I had to write a long explanation, going down to details very similar to
those RGB raised here
(but not with the same sharp humor), to justify buying a cluster as opposed
to a
"turnkey" solution akin to this "deskside supercomputer" just two weeks ago.
I went through the same arguments that RGB used here: environmental issues
(power, A/C, floor space), TCO,
sys admin and maintenance, pros and cons of COTS vs. proprietary HW and SW,
etc.
I won, but now I'll probably have to "dejavu it all over again",
when people here learn about the CX1.
Don't forget, one can get turnkey linux clusters too. Penguin/Scyld
would love to sell you one. Joe and Scalable informatics would cheerily
custom engineer you one. Or you can EASILY get "semi-turnkey" -- a
rackful of 1U systems all wired up and ready for you to just drop on
your own favorite OS image, which for DCHP/PXE installs is about as
difficult as, well, configuring a server with a repo and e.g. kickstart
file turning the nodes on.
If my boss was not knowledgeable in Physics and computing,
I would have to invite RGB to come here to give my boss a briefing.
And to recommend buying a beer keg refrigerator along with a cluster, of
course.
And if you (or rather he) paid me vast sums of money, I'd be happy to
come give him one, keg and all. However, if I didn't charge $250+/hour
plus travel and expenses for my time, your boss wouldn't listen to me
anyway, and baby needs a new pair of shoes... because he's away in
college and shoes, they are EXPENSIVE in college.

I do think we're on the track of something important here, though. A
beer-cooled cluster. This is brilliant! Why hasn't anyone thought of
it before? They make half- and quarter-keg sized refrigerators designed
to keep a liquid at a steady 40F temperature. Beer contains alcohol,
sort of like antifreeze only "less toxic" to the environment, um, acts
like a preservative or something. The bubbles, lessee, they act as
shock absorbers against thermal expansion as the cold beer absorbs the
CPU's heat and returns to the keg for recooling.

Best of all, when a node breaks the beer is much cheaper than
antidepressants and therapy for the person who comes to replace it (and
hey, one has to drain the entire system when that occurs and refill it
with fresh beer, and one hates to waste anything, right?:-)

rgb
Gus Correa
--
Robert G. Brown Phone(cell): 1-919-280-8443
Duke University Physics Dept, Box 90305
Durham, N.C. 27708-0305
Web: http://www.phy.duke.edu/~rgb
Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php
Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977
Robert G. Brown
2008-09-18 18:46:38 UTC
Permalink
Post by Mark Hahn
Post by Robert G. Brown
* Cluster people with significant constraints on space, power, or AC.
just space, really. blade systems used to be almost unique in offering
high-efficiency power solutions, but I think most or all that's become
available in the commodity market now. (that is, 80-90% psu's in normal
1U servers).
and remember, less space and approximately the same power means higher
heat-density. I've never seen a lot of fully populated blade enclosures
in one spot (which is kinda the point), though it should be doable with
rack-back heat exchangers.
actually, are there any blade systems that skip air-cooling entirely?
that would actually make sense - if you're going to go for bespoke power
because of potentially greater efficiency, bespoke cooling makes sense
for the same reason.
Post by Robert G. Brown
* businesses that want a turnkey system, typically for HA
applications, that is compact and "easy" to support.
that part never made sense to me. I'm skeptical that the management
interface for blade systems is better than plain old IPMI. prettier,
perhaps.
Agreed and agreed, but there it is. If nothing else, a small system
that hides its "rackiness" LOOKS easier to manage than a rack of 1U or
3U boxes. And I admit I don't know the MTBF numbers and couldn't tell
you if they are more reliable or less expensive to manage. However,
they never quite go away, so somebody keeps buying them... and I doubt
that a lot are bought by clusterheads. ;-)
Post by Mark Hahn
Post by Robert G. Brown
And that is fair enough, actually. Some places one literally has a
closet to put one's cluster in, and if one populates the closet with a
a closet which just happens to have a huge jet of cold air going through it...
http://www.cray.com/Products/CX1/Product/Specifications.aspx
claims 1600W, 92% efficient. their pages don't give much info on the
engineering of the blades, though. given that you have to add ipmi
as an option card, it looks pretty close to commodity parts to me.
1600W for 8 8-core, 300 MHz, 2 GB/core RAM blades at full computational
load?

I don't believe it. I don't think my laptop averages that little (25W)
per core, sustained, using clock shifting and sitting around idle a lot.
It is sitting on my lap at the moment and is burning my hands on the
wrist-rests and my legs through my pants just a little, all the time, at
800 MHz idle except for typing.

Somebody just mailed me specs offline that suggested 375W/card, which is
at least not completely unreasonable (although I'd want to see
Kill-a-Watt validated wall-power draw, not a number that might be some
sort of "average" or might refer to idle power of their slowest clock
system. My KaW shows anywhere from 20-40% power variability from idle
to load in many systems, and the power drawn by the CPU, or a minimally
configured motherboard isn't the same as that drawn by a full system
including heat losses in the power supply etc.

1600W sounds not unreasonable for a >>4<< blade system, and is still a
wee bit warm for MY office, but one might be able to plug it into a 20A
circuit and not instantly blow the breaker. And with 3 blades, or 2, it
would still have 16-24 cores -- a very respectable total. But then one
could match it with a couple or three towers, which would also have
about the same footprint (except for height) and would almost certainly
cost only half as much.

rgb
--
Robert G. Brown Phone(cell): 1-919-280-8443
Duke University Physics Dept, Box 90305
Durham, N.C. 27708-0305
Web: http://www.phy.duke.edu/~rgb
Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php
Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977
Mark Hahn
2008-09-18 19:15:19 UTC
Permalink
Post by Robert G. Brown
Post by Mark Hahn
http://www.cray.com/Products/CX1/Product/Specifications.aspx
claims 1600W, 92% efficient. their pages don't give much info on the
this is just for 1 psu, and at least 2 are needed.
Post by Robert G. Brown
1600W for 8 8-core, 300 MHz, 2 GB/core RAM blades at full computational
load?
sorry, my post was misleading. it's 3200 for 64c, 200W/socket. these
sockets are 80W TDP (50W option), so that seems pretty believable.
Post by Robert G. Brown
Somebody just mailed me specs offline that suggested 375W/card, which is
I see up to about 350W with my power meter on a commodity (HP) dual
2.67 intel quad-core system. (1 disk, 8GB non-fbdimm memory).
Matt Lawrence
2008-09-19 21:05:20 UTC
Permalink
Post by Mark Hahn
and remember, less space and approximately the same power means higher
heat-density. I've never seen a lot of fully populated blade enclosures
in one spot (which is kinda the point), though it should be doable with
rack-back heat exchangers.
I'm actually looking for pointers to rear door heat exchangers. From
reading the Liebert spec on theirs, it seems to only handle about
8KW/rack. Since it's fairly easy to exceed that with modern systems, I'm
very curious to know what else is out there. At this point, I suspect
that a fair amount of room cooling may be needed as well.

-- Matt
It's not what I know that counts.
It's what I can remember in time to use.
Greg Lindahl
2008-09-19 21:07:05 UTC
Permalink
Post by Matt Lawrence
I'm actually looking for pointers to rear door heat exchangers.
Lux, James P
2008-09-18 15:59:10 UTC
Permalink
Post by Lux, James P
-----Original Message-----
Sent: Thursday, September 18, 2008 7:22 AM
To: Gus Correa
Cc: Beowulf
Subject: Re: [Beowulf] Re: MS Cray
Post by Gus Correa
After I configured it with eight dual-slot quad-core Xeon E5472
(3.0GHz) compute nodes, 2GB/core RAM, IPMI, 12-port DDR IB switch
(their smallest), MS Windows installed, with one year standard 9-5
support, and onsite installation, the price was over $82k.
It sounds pricey to me, for an 8 node cluster.
Storage or viz node choices, 24-port IB to connect to other
enclosures, etc, are even more expensive.
Again, excellently well put. This is literally the bottom
line. What we are really talking about is form factor and
who does what. People usually are pretty careful with their
money, at least within their range of knowledge. When bladed
systems first started coming out -- which was many years ago
at this point -- I did a bit of an on-list CBA of them and
concluded that there was a price premium of something like a
factor of 2 for them, compared to the price of an equivalent
stack of rackmounted nodes, more like 3 compared to a shelf
full of tower units.
I asked "why would anyone pay that"?
<snip of rgb's excellent description of infrastructure issues>
Post by Lux, James P
This little exercise in the realities of infrastructure
planning exposes the fallacy of the "desktop cluster" in MOST
office environments, including research miniclusters in a lot
of University settings. There exist spaces -- perhaps big
labs, with their own dedicated climate control and lots of
power -- where one could indeed plug right in and run, but
your typical office or cubicle is not one of them. Those
same spaces have room for racks, of course, if they have room
for a high density blade chassis.
If you already have, or commit to building, an infrastructure
space with rack room, real AC, real power, you have to look
SERIOUSLY at whether you want to pay the price premium for
small-form factor solutions. But that premium is a lot
smaller than it was eight or so years ago, and there ARE
places with that proverbial broom closet or office that is
the ONLY place one can put a cluster. For them, even with
the relatively minor renovations needed to handle 3-4 KW in a
small space, it might well be worth it.
I suspect that there is some non-negligible demand for these boxes, notwithstanding the high cost. (esp viewed in terms of keeping the mfr line for the product going.. Not like either Cray or MS is depending on these sales to keep the company alive)

How about as an "executive toy" for the guy in the corner office running financial models? (I am a Master of the Universe, and I must have my special data entirely under my control.)

How about in places where the organizational pain that comes with being in the "machine room" is high? (All systems in the main computer room shall be under the cognizance of Senior VicePresident of MachineRoom Operations Smith. SVP Smith dictates: All systems in the machine room shall be made available to all users so as to efficiently allocate computational resources, since my bonus depends on reducing the metric of "idle time percentage". SVP of IT Security Jones: All shared computational resources shall use the corporate standard software disk encryption and must run both McAfee and Symantec AntiVirus in continuous scan mode. SVP of Network Management Wilson: In order to achieve maximum commonality and facilitate continuing reuse of computing assets purchased in 1991, all computers shall provide a connection of 10Base2 Ethernet at 2 Mbps. SVP of CustomerProprietaryInformationSecurity Brown: All systems in the machineroom shall use the corporate secure SAN. And so it goes..)


I think the model that John Vert mentioned, using it as a software development workstation to try things out before running on the "big iron" is actually probably a more likely scenario. And for that, you might not want the full up configuration, just enough to make it a "real" cluster so you can work out the interprocessor communications issues.

Jim
John Hearns
2008-09-18 16:24:43 UTC
Permalink
Post by Lux, James P
How about as an "executive toy" for the guy in the corner office running
financial models? (I am a Master of the Universe, and I must have my special
data entirely under my control.)
Cough. I think a lot of them just left the building, carrying golf clubs (as
pictured in trhe evening free newspapers in London on Monday).
Lux, James P
2008-09-18 16:30:15 UTC
Permalink
2008/9/18 Lux, James P <***@jpl.nasa.gov>
How about as an "executive toy" for the guy in the corner office running financial models? (I am a Master of the Universe, and I must have my special data entirely under my control.)
John Hearns [***@googlemail.com]

Cough. I think a lot of them just left the building, carrying golf clubs (as pictured in trhe evening free newspapers in London on Monday).

---

Well, there *is* that aspect.. So there might be a temporary hiccup in demand, at least in that market. On the other hand, my wife cynically comments that they'll need a supercomputer to unscramble the mess.

Jim
Robert G. Brown
2008-09-18 19:06:11 UTC
Permalink
Post by Lux, James P
I think the model that John Vert mentioned, using it as a software
development workstation to try things out before running on the "big
iron" is actually probably a more likely scenario. And for that, you
might not want the full up configuration, just enough to make it a
"real" cluster so you can work out the interprocessor communications
issues.
Which is fine, if the "big iron" (more likely "big bad wulf":-) you want
is running the same interconnect, etc. In which case as few as two
blades might work, but then, so would two towers at about the same
footprint (in all dimensions) and half the cost.

A bladed system full, well, that makes sense of a sort as it is the only
way to get that density, sort of, except as Mark pointed out the
advantage isn't quite as pronounced as perhaps it once was. A bladed
system with two or three blades and five or six empty slots? That's
just plain dumb. Even my desk, swollen as it is with the flood of
papers I'll get around looking at the day Satan goes snow skiing down
the slopes of hell, can be trivially rearranged to add two more
mid-towers to the one that is already there and a small network
interface. In fact, I've got three mid-tower chassis (two of them dead)
sitting in my office anyway. I could probably build myself a 17 core
development cluster out of spare parts and a couple of new
motherboard/CPU/memory sets for $4K, or spend even less on just single
quad motherboards and a better network for 9 cores.

As is all too often true, a niche product has narrow boundaries on the
places it SANELY makes sense, although naturally that is almost
irrelevant to the issue of marketing it. And hey, Penguin's "blade
center" is awfully pretty and awfully high density if one ever could
afford to buy it fully populated.

One of my earliest personal cases of optimism was buying an $80K SGI
220S (think of it as a "very early bladed server" the size of a
refrigerator and drawing anywhere from 1-3 KW) with just one dual CPU
blade, thinking we'd eventually get more at $20-30K/blade. Ha ha ha.
We sold it six or so years later for $3000, when we could get over the
counter Sparcstations that were STILL expensive at $5K or thereabouts
but which were even faster.

But people do buy them -- grab a copy of PC Mag or Linux Mag (now
sorta-defunct?) or whatever and leaf it. What do you find? Lots of ads
for rackmounts. A few ads for consumer systems (but not that many
anymore, outside of laptops). And yeah, quite a few ads for
bladeservers.

Just not USUALLY for cluster computing, I don't think, at least by
people on this list, which is a last bastion for non-turnkey beowulfery
where we always ask ourselves if it is better to buy a prettier and
niftier system or get more processors, and almost invariably select
"more processors" in an uglier form factor when we find it.

:-)

rgb
Post by Lux, James P
Jim
--
Robert G. Brown Phone(cell): 1-919-280-8443
Duke University Physics Dept, Box 90305
Durham, N.C. 27708-0305
Web: http://www.phy.duke.edu/~rgb
Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php
Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977
Douglas Eadline
2008-09-21 15:37:15 UTC
Permalink
About Linux Magazine. The publication has become entirely
web based. All content is free after registration. The paper
subscriptions were absorbed by Linux Pro Magazine. There will
be more HPC content coming as well. I have been writing a weekly
opinion/rant piece since the beginning of the year.
There is an even HPC sections and newsletter.
HPC content lives in two places on the site:

http://www.linux-mag.com/solutions/hpc
http://www.linux-mag.com/resource-centers/todays-hpc-clusters

--
Doug
Post by Robert G. Brown
But people do buy them -- grab a copy of PC Mag or Linux Mag (now
sorta-defunct?) or whatever and leaf it. What do you find? Lots of ads
for rackmounts. A few ads for consumer systems (but not that many
anymore, outside of laptops). And yeah, quite a few ads for
bladeservers.
--
Doug
k***@neuralbs.com
2008-09-18 20:09:15 UTC
Permalink
Post by Lux, James P
Post by Lux, James P
-----Original Message-----
Sent: Thursday, September 18, 2008 7:22 AM
To: Gus Correa
Cc: Beowulf
Subject: Re: [Beowulf] Re: MS Cray
Post by Gus Correa
After I configured it with eight dual-slot quad-core Xeon E5472
(3.0GHz) compute nodes, 2GB/core RAM, IPMI, 12-port DDR IB switch
(their smallest), MS Windows installed, with one year standard 9-5
support, and onsite installation, the price was over $82k.
It sounds pricey to me, for an 8 node cluster.
Storage or viz node choices, 24-port IB to connect to other
enclosures, etc, are even more expensive.
Again, excellently well put. This is literally the bottom
line. What we are really talking about is form factor and
who does what. People usually are pretty careful with their
money, at least within their range of knowledge. When bladed
systems first started coming out -- which was many years ago
at this point -- I did a bit of an on-list CBA of them and
concluded that there was a price premium of something like a
factor of 2 for them, compared to the price of an equivalent
stack of rackmounted nodes, more like 3 compared to a shelf
full of tower units.
I asked "why would anyone pay that"?
<snip of rgb's excellent description of infrastructure issues>
Post by Lux, James P
This little exercise in the realities of infrastructure
planning exposes the fallacy of the "desktop cluster" in MOST
office environments, including research miniclusters in a lot
of University settings. There exist spaces -- perhaps big
labs, with their own dedicated climate control and lots of
power -- where one could indeed plug right in and run, but
your typical office or cubicle is not one of them. Those
same spaces have room for racks, of course, if they have room
for a high density blade chassis.
If you already have, or commit to building, an infrastructure
space with rack room, real AC, real power, you have to look
SERIOUSLY at whether you want to pay the price premium for
small-form factor solutions. But that premium is a lot
smaller than it was eight or so years ago, and there ARE
places with that proverbial broom closet or office that is
the ONLY place one can put a cluster. For them, even with
the relatively minor renovations needed to handle 3-4 KW in a
small space, it might well be worth it.
I suspect that there is some non-negligible demand for these boxes,
notwithstanding the high cost. (esp viewed in terms of keeping the mfr
line for the product going.. Not like either Cray or MS is depending on
these sales to keep the company alive)
How about as an "executive toy" for the guy in the corner office running
financial models? (I am a Master of the Universe, and I must have my
special data entirely under my control.)
How about in places where the organizational pain that comes with being in
the "machine room" is high? (All systems in the main computer room shall
be under the cognizance of Senior VicePresident of MachineRoom Operations
Smith. SVP Smith dictates: All systems in the machine room shall be made
available to all users so as to efficiently allocate computational
resources, since my bonus depends on reducing the metric of "idle time
percentage". SVP of IT Security Jones: All shared computational resources
shall use the corporate standard software disk encryption and must run
both McAfee and Symantec AntiVirus in continuous scan mode. SVP of
Network Management Wilson: In order to achieve maximum commonality and
facilitate continuing reuse of computing assets purchased in 1991, all
computers shall provide a connection of 10Base2 Ethernet at 2 Mbps. SVP of
CustomerProprietaryInformationSecurity Brown: All systems in the
machineroom shall use the corporate secure SAN. And so it g!
oes..)
Yupp, that's where we are at and it's why they hire students like me to
take care of the dep's isolated server room because most the SVPs think
HPC == HA (clustering).
Post by Lux, James P
I think the model that John Vert mentioned, using it as a software
development workstation to try things out before running on the "big iron"
is actually probably a more likely scenario. And for that, you might not
want the full up configuration, just enough to make it a "real" cluster so
you can work out the interprocessor communications issues.
Jim
If that is the case, just get a multi-core system with Linux. And if you
like FluFF (and treasure your auditory capabilities), get a Power Mac.

Eric
Continue reading on narkive:
Loading...