[Beowulf] If I were specifying a new custer...

[Douglas Eadline]

All:

Over the last several months I have been reading about:

1) Spectre/meltdown
2) Intel Fab issues
3) Supermicro MB issues

I started thinking, if I were going to specify a
single rack cluster, what would I use?

I'm assuming a general HPC workload (not deep learning or
analytics) I need to choose Xeon/Epyc, IB/Omni,
Lustre/Ceph/BeGFS, should all nodes have GPU?

I'm interested what members of this list think?

[Benson Muite]

Post by Douglas Eadline
1) Spectre/meltdown
2) Intel Fab issues
3) Supermicro MB issues
I started thinking, if I were going to specify a
single rack cluster, what would I use?
I'm assuming a general HPC workload (not deep learning or
analytics) I need to choose Xeon/Epyc, IB/Omni,
Lustre/Ceph/BeGFS, should all nodes have GPU?
I'm interested what members of this list think?

Check back for the winning bid here:

https://www.ictp.it/about-ictp/open-bids.aspx

_______________________________________________
Beowulf mailing list, ***@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www

[Scott Atchley]

What do your apps need?

• Lots of memory?

Perhaps Power9 or Naples with 8 memory channels? Also, Cavium ThunderX2.

• More memory bandwidth?

Same as above.

• Max single thread performance?

Intel or Power9?

• Are your apps GPU enabled? If not, do you have budget/time/expertise to
do the work?

If not, then stick with CPUs.

• Need lots of PCIe for accelerators, SSDs, NICs?

Naples.

• Interconnect?

If Intel, either of the above. Does OPA work with Naples (I don't know)? If
Power/ARM and possibly Naples, then IB.

Sorry to give the default computer science answer of "It depends."

Scott

Post by Douglas Eadline
1) Spectre/meltdown
2) Intel Fab issues
3) Supermicro MB issues
I started thinking, if I were going to specify a
single rack cluster, what would I use?
I'm assuming a general HPC workload (not deep learning or
analytics) I need to choose Xeon/Epyc, IB/Omni,
Lustre/Ceph/BeGFS, should all nodes have GPU?
I'm interested what members of this list think?
--
Doug
--
MailScanner: Clean
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf

[Chris Samuel]

Post by Scott Atchley
Perhaps Power9 or Naples with 8 memory channels? Also, Cavium ThunderX2.

I'm not sure if Power or ARM (yet) qualify for a general HPC workload
that Doug mentions; sadly a lot of the commercial codes are only
available for x86-64 these days. MATLAB dropped PowerPC support back in
2007 for instance.

All the best,
Chris (still in the UK)

[Jonathan Engwall]

You can narrow things down for instance you could start with warranty and support. Location is a factor too. You want every kind of stability, reliable power, cold water and to be near your suppliers, factoring in airports or interstate highways.

Post by Scott Atchley
Perhaps Power9 or Naples with 8 memory channels? Also, Cavium ThunderX2.

I'm not sure if Power or ARM (yet) qualify for a general HPC workload
that Doug mentions; sadly a lot of the commercial codes are only
available for x86-64 these days. MATLAB dropped PowerPC support back in
2007 for instance.

All the best,
Chris (still in the UK)

[John Hearns via Beowulf]

Doug,
I think the ARM/Cavium Thunder is going to see a lot of attention.
I saw a report recently from the Bristol/Cray Brunel cluster - they are
offering a range of chemistry codes and OpenFOAM,
compiled up for ARM.
Poke me and I will search for the report - I saw it on a twitter feed.

Regarding the Supermicro MB issue, I think there is egg on the face here.
Apple have robustly denied this,
and things seem to have gone very quiet.
Regarding the Intel Fab issues, what is the public evidence for this?
I guess there may be a lot of industry scuttlebutt around.

My own thoughts on this are that as I always say - in IT follow the herd,
or you will be trampled underfoot.
By that I mean that it is intellectually satisfying to discover a new
architecture/bit transport/programming language which fits exactly to your
problem
and gives a huge increase in time to solution.
Yes, grab onto this and implement it in the short term but dont bet on it
being around for long term (save programming languages)
Look at Beowulfery itself - you are staging a celebration of 20 years of
Beowulfery at SC. It is of course COTS components applied to what
at the time was a field populated by esoteric architectures.
From my own experience, look at ATM networking. I put a lot of effort into
learning and managing ATM networks. Technically it is superior.
But development stalled at 622Mbps and with too few companies supplying kit
there was no effort in writing drivers for OS updates.
So we see the ubiquity of Ethernet - or what at the high end probably
pretends to be Ethernet (I am referring to the zero loss RDMA networks).

What I am saying is the next generation will be that which is created by
the Web Scale generation. Kubernetes clusters running CoreOS.
Containers - and we already have seen comments from our friends in
Singularity that the Open Container Initiative is basically a copy of
Docker and is not appropriate for HPC (this is another discussion to be
had).
I will now be controversial. The choice of CPUs / GPUs/ TPUs/ FPGAs or any
accelerators in your next generation cluster
will be the ones chosen for use by Amazon (Azure, Google, Baidu....). these
are the ones where the industry will be putting in the effort - as they are
the biggest market.
Remember - run with the herd or you will be trampled underfoot.


(ps. Is this worthy of a Clustermonkey blog article? More like a blog
rant...)


On Fri, 12 Oct 2018 at 03:18, Jonathan Engwall <

Post by Jonathan Engwall
You can narrow things down for instance you could start with warranty and
support. Location is a factor too. You want every kind of stability,
reliable power, cold water and to be near your suppliers, factoring in
airports or interstate highways.

Post by Scott Atchley
Perhaps Power9 or Naples with 8 memory channels? Also, Cavium ThunderX2.

I'm not sure if Power or ARM (yet) qualify for a general HPC workload
that Doug mentions; sadly a lot of the commercial codes are only
available for x86-64 these days. MATLAB dropped PowerPC support back in
2007 for instance.
All the best,
Chris (still in the UK)
--
Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf

[Gerald Henriksen]

Post by John Hearns via Beowulf
I think the ARM/Cavium Thunder is going to see a lot of attention.
I saw a report recently from the Bristol/Cray Brunel cluster - they are
offering a range of chemistry codes and OpenFOAM,
compiled up for ARM.
Poke me and I will search for the report - I saw it on a twitter feed.

ARM essentially has 2 problems.

One, you go from cheap/limited SOC boards like the Raspberry Pi and
jump straight into the expensive Cavium line. There is no affordable
ARM option for developers to use painfree to write code, port code,
and test code.

If ARM, or Power, want to move from their current positions in the
market they really need to provide affordable developer machines,
machines priced at a point where the open source community can justify
buying it as a secondary machine to work on making the Linux
distributions more solid and the associated software.

Two, for whatever reason the only company that seems to be able to
produce ARM processors with decent peformance is Apple, and they
aren't giving up their advantage. Microsoft would like to move into
the ARM based notebook market but so far Qualcomm doesn't seem to be
able to provide a chip with adequate performance. The ARM ecosystem
needs to do much better at converting the ARM designs into actual
hardware.

Post by John Hearns via Beowulf
Regarding the Intel Fab issues, what is the public evidence for this?
I guess there may be a lot of industry scuttlebutt around.

Earning reports, stock downgrades, prices.

Intel is having serious issues moving from 14nm to 10nm, is 3 years
late and now saying sometime in 2019, and is having supply issues
given they have fabs unable to supply product as the conversion
continues to be extended.

_______________________________________________
Beowulf mailing list, ***@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mai

[Joe Landman]

Post by Gerald Henriksen

Post by John Hearns via Beowulf
I think the ARM/Cavium Thunder is going to see a lot of attention.
I saw a report recently from the Bristol/Cray Brunel cluster - they are
offering a range of chemistry codes and OpenFOAM,
compiled up for ARM.
Poke me and I will search for the report - I saw it on a twitter feed.

ARM essentially has 2 problems.

I'd say 3, including what you wrote.

#3  End users are generally loathe to re-compile applications for a new
processor/architecture, unless it gives them substantial benefit.  GPU
rewrites were virtually guaranteed, once people got over the learning
curve, as early (minimal) efforts yielded 5-10x performance bumps.  More
work, and a rethinking of the application gave significant benefit.

ARM doesn't and as far as I can tell, won't have this advantage. The
only advantage it brings potentially is power consumption per cycle. 
And this advantage evaporates once you start looking at the high
computational power chips.

Recycling an old joke on this, ARM is the CPU of the future, and always
will be.

Its not ABI compatible, ISA compatible.  Its not "blow the doors off"
faster.  Its not (in the performance configurations) lower power.

Exactly what is the market draw of these processors?  What niche are
they seeking to fill, and what unique advantages does it bring? These
are not apparent.

Just my thoughts, but I've worked with some ARM product builders in the
past, and have been burned by the misalignment between reality and rhetoric.
--
Joe Landman
e: ***@gmail.com
t: @hpcjoe
w: https://scalability.org
g: https://github.com/joelandman
l: https://www.linkedin.com/in/joelandman

_______________________________________________
Beowulf mailing list, ***@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) v

[Chris Samuel]

Post by Gerald Henriksen
If ARM, or Power, want to move from their current positions in the
market they really need to provide affordable developer machines,

Not sure if this comes in at a price point that makes sense for this, but
there is now an ATX Power9 mainboard available.

https://raptorcs.com/TALOSIILITE/

They claim:

https://twitter.com/RaptorCompSys/status/1020371675316215809

# TalosIILite in stock and ready to ship! #POWER9 mainboard + CPU + RAM + HSF
# for under $2,000 USD, what's not to like? Supports all of our Sforza CPU
# options, from 4 core to the high end 22 core CPUs.

All the best,
Chris

[Gerald Henriksen]

Post by Chris Samuel

Post by Gerald Henriksen
If ARM, or Power, want to move from their current positions in the
market they really need to provide affordable developer machines,

Not sure if this comes in at a price point that makes sense for this, but
there is now an ATX Power9 mainboard available.
https://raptorcs.com/TALOSIILITE/
https://twitter.com/RaptorCompSys/status/1020371675316215809
# TalosIILite in stock and ready to ship! #POWER9 mainboard + CPU + RAM + HSF
# for under $2,000 USD, what's not to like? Supports all of our Sforza CPU
# options, from 4 core to the high end 22 core CPUs.

Not really.

While there obviously is a lot of corporate funded work in the open
source community I would guess little of it is interested in anything
but the traditional AMD/Intel systems, and maybe ARM.

To give a new / minor platform traction you really need to have
something priced where it can be a personal purchase, normally as a
secondary machine as few people will move to ARM or Power (at this
point) as an only machine.

They do however have another platform coming, no prices yet, but
should hopefully be more affordable in the Blackbird:

https://wiki.raptorcs.com/wiki/Blackbird

And to point out what is obvious to many, the reason these cheaper
systems are needed is to get all that open source software working and
tested, Raptor has had to do work just to get Chrome working on Power:

https://www.phoronix.com/scan.php?page=news_item&px=Raptor-Chrome-JIT-PPC64LE-Work
_______________________________________________
Beowulf mailing list, ***@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listin

[Gerald Henriksen]

Post by John Hearns via Beowulf
Doug,
I think the ARM/Cavium Thunder is going to see a lot of attention.
I saw a report recently from the Bristol/Cray Brunel cluster - they are
offering a range of chemistry codes and OpenFOAM,
compiled up for ARM.

Perhaps this paper?
http://uob-hpc.github.io/assets/cug-2018.pdf

Also interesting is another entrant into the ARM server market is
Ampere, who have had some hardware tested by Phoronix:

https://www.phoronix.com/scan.php?page=article&item=ampere-emag-osprey&num=1

Perhaps more importantly though is they appear to be a bit more
developer / open source community friendly with an employee commenting
in the Phoronix discussion thread and they even have a developer
website:

https://developer.amperecomputing.com/
_______________________________________________
Beowulf mailing list, ***@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit

[Jonathan Engwall]

"Run with the heard" great quote right there.
☺
Something like: IaaS IBM financial services with a hybrid for prediction and storage...Maybe no storage promise at all.

Post by Gerald Henriksen

Post by John Hearns via Beowulf
I think the ARM/Cavium Thunder is going to see a lot of attention.
I saw a report recently from the Bristol/Cray Brunel cluster - they are
offering a range of chemistry codes and OpenFOAM,
compiled up for ARM.
Poke me and I will search for the report - I saw it on a twitter feed.

ARM essentially has 2 problems.

I'd say 3, including what you wrote.

#3  End users are generally loathe to re-compile applications for a new
processor/architecture, unless it gives them substantial benefit.  GPU
rewrites were virtually guaranteed, once people got over the learning
curve, as early (minimal) efforts yielded 5-10x performance bumps.  More
work, and a rethinking of the application gave significant benefit.

ARM doesn't and as far as I can tell, won't have this advantage. The
only advantage it brings potentially is power consumption per cycle. 
And this advantage evaporates once you start looking at the high
computational power chips.

Recycling an old joke on this, ARM is the CPU of the future, and always
will be.

Its not ABI compatible, ISA compatible.  Its not "blow the doors off"
faster.  Its not (in the performance configurations) lower power.

Exactly what is the market draw of these processors?  What niche are
they seeking to fill, and what unique advantages does it bring? These
are not apparent.

Just my thoughts, but I've worked with some ARM product builders in the
past, and have been burned by the misalignment between reality and rhetoric.
--
Joe Landman
e: ***@gmail.com
t: @hpcjoe
w: https://scalability.org
g: https://github.com/joelandman
l: https://www.linkedin.com/in/joelandman

_______________________________________________
Beowulf mailing list, ***@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, ***@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailm

[Douglas Eadline]

Thanks for all the replies. I may write this up.

One other pointed question.

In general, how much has the Spectere/meltdown issue and Intel
Fab issue effected your decisions of Intel vs AMD?

Or are your decisions still based on performance/price/other
benchmarks?

--
Doug

Post by Douglas Eadline
1) Spectre/meltdown
2) Intel Fab issues
3) Supermicro MB issues
I started thinking, if I were going to specify a
single rack cluster, what would I use?
I'm assuming a general HPC workload (not deep learning or
analytics) I need to choose Xeon/Epyc, IB/Omni,
Lustre/Ceph/BeGFS, should all nodes have GPU?
I'm interested what members of this list think?
--
Doug
--
MailScanner: Clean
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf

[Andrew Latham]

I know a lot of organizations that are further building out their walled
gardens and securing the interaction with the Internet instead of acting on
the Spectre/Vulndejour by switching vendor.

Post by Douglas Eadline
Thanks for all the replies. I may write this up.
One other pointed question.
In general, how much has the Spectere/meltdown issue and Intel
Fab issue effected your decisions of Intel vs AMD?
Or are your decisions still based on performance/price/other
benchmarks?
--
Doug

Post by Douglas Eadline
1) Spectre/meltdown
2) Intel Fab issues
3) Supermicro MB issues
I started thinking, if I were going to specify a
single rack cluster, what would I use?
I'm assuming a general HPC workload (not deep learning or
analytics) I need to choose Xeon/Epyc, IB/Omni,
Lustre/Ceph/BeGFS, should all nodes have GPU?
I'm interested what members of this list think?
--
Doug
--
MailScanner: Clean
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf

--
Doug
--
MailScanner: Clean
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf

[Lachlan Musicman]

Post by Andrew Latham
I know a lot of organizations that are further building out their walled
gardens and securing the interaction with the Internet instead of acting on
the Spectre/Vulndejour by switching vendor.

Post by Douglas Eadline
In general, how much has the Spectere/meltdown issue and Intel
Fab issue effected your decisions of Intel vs AMD?

Yes, we don't buy hardware often enough to warrant taking an action like
that. But we always know that our security could do with more work, and
took it as an opportunity to do so. By which I mean we invested more
resources, but also successfully used the Spectre spectre on our management
as evidence that security needed more resources.

Which is always nice.

Cheers
L.

------
'...postwork futures are dismissed with the claim that "it is not in our
nature to be idle", thereby demonstrating at once an essentialist view of
labor and an impoverished imagination of the possibilities of nonwork.'

Kathi Weeks, *The Problem with Work: Feminism, Marxism, Antiwork Politics
and Postwork Imaginaries*
<https://www.dukeupress.edu/The-Problem-with-Work/>

[Jonathan Engwall]

Post by Gerald Henriksen

Post by Chris Samuel

Post by Gerald Henriksen
If ARM, or Power, want to move from their current positions in the
market they really need to provide affordable developer machines,

Not sure if this comes in at a price point that makes sense for this, but
there is now an ATX Power9 mainboard available.
https://raptorcs.com/TALOSIILITE/
https://twitter.com/RaptorCompSys/status/1020371675316215809
# TalosIILite in stock and ready to ship! #POWER9 mainboard + CPU + RAM + HSF
# for under $2,000 USD, what's not to like? Supports all of our Sforza CPU
# options, from 4 core to the high end 22 core CPUs.

Not really.
While there obviously is a lot of corporate funded work in the open
source community I would guess little of it is interested in anything
but the traditional AMD/Intel systems, and maybe ARM.
To give a new / minor platform traction you really need to have
something priced where it can be a personal purchase, normally as a
secondary machine as few people will move to ARM or Power (at this
point) as an only machine.
They do however have another platform coming, no prices yet, but
https://wiki.raptorcs.com/wiki/Blackbird
And to point out what is obvious to many, the reason these cheaper
systems are needed is to get all that open source software working and

Two nights ago I turned two lines of core python into seven working lines. It is extremely frustrating. Lately I just expect it.

Post by Gerald Henriksen
https://www.phoronix.com/scan.php?page=news_item&px=Raptor-Chrome-JIT-PPC64LE-Work
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

[John Hearns via Beowulf]

On the subject of new architectures, James Lin from the Shanghai HPC Centre
tweeted this:
In HPC China2018, I see the prototype node of Tianhe3. Processor
(FT-matrix 2000+): 128 cores, 2TFlops in DP --> Node: 3 processors, 6TF -->
The prototype system: 512 nodes, 3PFlops. The interconnection is
200Gbps. Details will be available online later this month. Looking at
the Matrix processor it is very interesting:
https://en.wikichip.org/wiki/nudt/matrix-2000 Each Matrix consist for four
'super nodes' with 32 cores and its own paths to DDR4 memory. So you would
imagine there are 4 separate chips in the package, but I may be wrong. Each
core has two 256bit wide vector units. Clocked at 1.2Ghz which makes sense.
A couple of observations: A single 128 CPU machine with 256 wide vector
units would have been a pretty respectable supercomputer in itself not so
long ago. If I have not miscalculated, it would have got you near the top
of the Top 500 in 2010 I wish I could get a hold of one, or a test drive -
probably not much hope of that. Stu Midgley - I think you need to get your
order in for a boat load of these! So I will stick my neck out - Doug, get
on a flight to Shanghai for your next clustermonkey building blocks.






On Thu, 18 Oct 2018 at 03:24, Jonathan Engwall <

Post by Jonathan Engwall

Post by Gerald Henriksen

Post by Chris Samuel

Post by Gerald Henriksen
If ARM, or Power, want to move from their current positions in the
market they really need to provide affordable developer machines,

Not sure if this comes in at a price point that makes sense for this,

but

Post by Gerald Henriksen

Post by Chris Samuel
there is now an ATX Power9 mainboard available.
https://raptorcs.com/TALOSIILITE/
https://twitter.com/RaptorCompSys/status/1020371675316215809
# TalosIILite in stock and ready to ship! #POWER9 mainboard + CPU + RAM

+ HSF

Post by Gerald Henriksen

Post by Chris Samuel
# for under $2,000 USD, what's not to like? Supports all of our Sforza

CPU

Post by Gerald Henriksen

Post by Chris Samuel
# options, from 4 core to the high end 22 core CPUs.

Not really.
While there obviously is a lot of corporate funded work in the open
source community I would guess little of it is interested in anything
but the traditional AMD/Intel systems, and maybe ARM.
To give a new / minor platform traction you really need to have
something priced where it can be a personal purchase, normally as a
secondary machine as few people will move to ARM or Power (at this
point) as an only machine.
They do however have another platform coming, no prices yet, but
https://wiki.raptorcs.com/wiki/Blackbird
And to point out what is obvious to many, the reason these cheaper
systems are needed is to get all that open source software working and

Two nights ago I turned two lines of core python into seven working lines.
It is extremely frustrating. Lately I just expect it.
https://www.phoronix.com/scan.php?page=news_item&px=Raptor-Chrome-JIT-PPC64LE-Work

Post by Gerald Henriksen
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit

http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf

[Stu Midgley]

he he he... but so many phi's at 3.5/7 TFlops... when they automagically
speed boost to 1.6GHz if you can cool them...


Stu Midgley - I think you need to get your order in for a boat load of

these! So I will stick my neck out - Doug, get on a flight to Shanghai for
your next clustermonkey building blocks.