[Beowulf] Problems with Dell M620 and CPU power throttling

[Bill Wichser]

Since January, when we installed an M620 Sandybridge cluster from Dell,
we have had issues with power and performance to compute nodes. Dell
apparently continues to look into the problem but the usual responses
have provided no solution. Firmware, BIOS, OS updates all are fruitless.

The problem is that the node/CPU is power capped. We first detected
this with the STREAM benchmark, a quick run, which shows memory
bandwidth around 2000 instead of the normal 13000 MB/s. When the CPU is
in the C0 state, this drops to around 600.

The effect appears randomly across the entire cluster with 5-10% of the
nodes demonstrating some slower performance. We don't know what
triggers this. Using "turbostat" we can see that the GHz of the cores
is >= 1 in most cases, dropping to about 0.2 in some of the worst cases.
Looking at the power consumption by either the chassis GUI or using
"impitool sdr list" we see that there is only about 80 watts being used.

We run the RH 6.x release and are up to date with kernel/OS patches.
All firmware is up to date. Chassis power is configured as
non-redundant. tuned is set for performance. Turbo mode is
on/hyperthreading is off/performance mode is set in BIOS.

A reboot does not change this problem. But a power cycle returns the
compute node to normal again. Again, we do not know what triggers this
event. We are not overheating the nodes. But while applications are
running, something triggers an event where this power capping takes effect.

At this point we remain clueless about what is causing this to happen.
We can detect the condition now and have been power cycling the nodes in
order to reset.

If anyone has a clue, or better yet, solved the issue, we'd love to hear
the solution!

Thanks,
Bill

[Mark Hahn]

Post by Bill Wichser
We run the RH 6.x release and are up to date with kernel/OS patches.

have you done any /sys tuning?

Post by Bill Wichser
non-redundant. tuned is set for performance. Turbo mode is

what knobs does tuned fiddle with? I would probably turn off all
auto-tuning and go strictly manual until the issue is understood.

Post by Bill Wichser
on/hyperthreading is off/performance mode is set in BIOS.

I found a "x86_energy_perf_policy.c" (author Len Brown) which
I run on my laptop to set powersave mode. it sets
MSR_IA32_ENERGY_PERF_BIAS. it says that the hardware default is performance,
but I wouldn't be surprised if "normal" is set by bios.

Post by Bill Wichser
A reboot does not change this problem. But a power cycle returns the
compute node to normal again. Again, we do not know what triggers this

unfortunately, a power cycle will also cool down the system,
so I don't see how it can be dissociated from heating.

Post by Bill Wichser
event. We are not overheating the nodes.

how do you know?

Post by Bill Wichser
But while applications are
running, something triggers an event where this power capping takes effect.

it might be interesting to examine the cpu-heatsink contact. what you're
describing could be explained by poor thermal HS contact (or poor HS flow).

or do you mean that you sample die temps at high resolution and know that
you're never hitting, say, 60C?

in some machines (albeit less often servers), acpi provides some knobs
which are made visible under /sys and seem to permit some control of
thermal mode (fan threshold or scaling).

Post by Bill Wichser
If anyone has a clue, or better yet, solved the issue, we'd love to hear
the solution!

what's your intake air temp? I would try giving it cold (say, 15C) air.

[Bill Wichser]

Of course we have done system tuning. Limits, arp table entries, block
size changes. tuned actually helped by setting a number of parameters
in sysctl among other sys parameters. Performance has increased due to
the changes made here. Prior to making these changes, when the system
was installed with a vanilla RH 6.1, we saw these problems. Incremental
changes helped performance but never resolved this power capping issue.

Instrumenting temperature probes on individual CPUs has not been
performed. When we look at temperatures from both the chassis and
ipmitool, we see no drastic peaks. Maybe we are getting a 60C peak that
we don't detect and that is the cause. But I doubt it.

Yes, a power cycle cools down the cores. Maybe that is the reset we
see. Prior to a power cycle we can look at the BMC temps, again using
ipmitool or the chassis GUI and wee that temps are well below 30C. We
also see that power consumption is around 80W. That tells me that the
system is cool enough. Should I not believe those values? i have no
reason to from past experience.

Input air is about 22C. For our data center, you'd have a better chance
of getting this adjusted to 15C than I would! As for fans, these don't
have any and are controlled at the chassis level.

For heat sink thermal grease problems, I'd expect this to be visible
using the ipmitools but maybe that is not where the temperatures are
being measured. I don't know about that issue. I'd expect that a bad
thermal grease issue would manifest itself by showing up on a per socket
level and not on both sockets. It seems odd that every node exhibiting
this problem would have both sockets having the same issue.

Again, the magnitude of the problem is about 5-10% at any time. Given
600 compute nodes, this is a lot of nodes showing similar problems. And
in most cases, a power cycle cures this issue.

Since we have not seen a node go "bad" while idle, this does point to
something overheating perhaps. The tools i know about to watch this
temperature are not sufficient in showing me this though.

Thanks,
Bill

Post by Mark Hahn

Post by Bill Wichser
We run the RH 6.x release and are up to date with kernel/OS patches.

have you done any /sys tuning?

Post by Bill Wichser
non-redundant. tuned is set for performance. Turbo mode is

what knobs does tuned fiddle with? I would probably turn off all
auto-tuning and go strictly manual until the issue is understood.

Post by Bill Wichser
on/hyperthreading is off/performance mode is set in BIOS.

I found a "x86_energy_perf_policy.c" (author Len Brown) which I run on
my laptop to set powersave mode. it sets
MSR_IA32_ENERGY_PERF_BIAS. it says that the hardware default is performance,
but I wouldn't be surprised if "normal" is set by bios.

Post by Bill Wichser
A reboot does not change this problem. But a power cycle returns the
compute node to normal again. Again, we do not know what triggers this

unfortunately, a power cycle will also cool down the system,
so I don't see how it can be dissociated from heating.

Post by Bill Wichser
event. We are not overheating the nodes.

how do you know?

Post by Bill Wichser
But while applications are
running, something triggers an event where this power capping takes effect.

it might be interesting to examine the cpu-heatsink contact. what you're
describing could be explained by poor thermal HS contact (or poor HS flow).
or do you mean that you sample die temps at high resolution and know
that you're never hitting, say, 60C?
in some machines (albeit less often servers), acpi provides some knobs
which are made visible under /sys and seem to permit some control of
thermal mode (fan threshold or scaling).

Post by Bill Wichser
If anyone has a clue, or better yet, solved the issue, we'd love to hear
the solution!

what's your intake air temp? I would try giving it cold (say, 15C) air.

[Mark Hahn]

Post by Bill Wichser
Of course we have done system tuning.

sorry for the unintentional condescenscion -
what I actually meant was "tuning of knobs located in /sys" :)

Post by Bill Wichser
Instrumenting temperature probes on individual CPUs has not been performed.
When we look at temperatures from both the chassis and ipmitool, we see no
drastic peaks. Maybe we are getting a 60C peak that we don't detect and that
is the cause. But I doubt it.

could you try "modprobe coretemp",
and see whether interesting things appear under:
/sys/devices/system/cpu/cpu*/thermal_throttle/core_throttle_count

afaik, reading the coretemp*/temp*_input values would let you do
higher-resolution monitoring to see whether you're getting spikes.

Post by Bill Wichser
power consumption is around 80W. That tells me that the system is cool
enough. Should I not believe those values? i have no reason to from past
experience.

I'm not casting aspersions, just that chassis temps don't tell the
whole story. is your exact model of CPU actually rated for higher
power? we've got some ProLiant SL230s Gen8 with E5-2680's - rated
for 130, and don't seem to be throttling.

Post by Bill Wichser
Input air is about 22C. For our data center, you'd have a better chance of
getting this adjusted to 15C than I would! As for fans, these don't have

yes, well, it is nice to have one's own datacenter ;)
but seriously, I find it sometimes makes a difference to open front
and back doors of the rack (if any), do some manual sampling of air
flow and temperatures (wave hand around)...

Post by Bill Wichser
For heat sink thermal grease problems, I'd expect this to be visible using
the ipmitools but maybe that is not where the temperatures are being
measured. I don't know about that issue. I'd expect that a bad thermal
grease issue would manifest itself by showing up on a per socket level and
not on both sockets. It seems odd that every node exhibiting this problem
would have both sockets having the same issue.

well, if both sockets have poor thermal contact with heatsinks...
I'm not trying to FUD up any particular vendor(s), but mistakes do happen.
I was imagining, for instance, that an assembly line might be set up
with HS and thermal compound tuned for E5-2637 systems (80W/socket),
but was pressed into service for some E5-2690 nodes (135W).

Post by Bill Wichser
Again, the magnitude of the problem is about 5-10% at any time. Given 600

if I understand you, the prevalence is only 5-10%, but the magnitude (effect)
is much larger, right?

regards, mark.

[Bill Wichser]

Post by Bill Wichser
Of course we have done system tuning.

sorry for the unintentional condescenscion - what I actually meant was
"tuning of knobs located in /sys" :)

Post by Bill Wichser
Instrumenting temperature probes on individual CPUs has not been
performed. When we look at temperatures from both the chassis and
ipmitool, we see no drastic peaks. Maybe we are getting a 60C peak
that we don't detect and that is the cause. But I doubt it.

could you try "modprobe coretemp", and see whether interesting things
/sys/devices/system/cpu/cpu*/thermal_throttle/core_throttle_count
afaik, reading the coretemp*/temp*_input values would let you do
higher-resolution monitoring to see whether you're getting spikes.

We have these already loaded and see values:

[***@r2c3n4 thermal_throttle]# ls
core_power_limit_count core_throttle_count package_power_limit_count
package_throttle_count
[***@r2c3n4 thermal_throttle]# cat *
18781048
0
18781097
0

This was what led us to how the chassis was limiting power. We had been
using redundancy and switched to non-redundant to try and eliminate. We
believe that we see these messages when the CPU is throttling up in
power.

[Mark Hahn]

Post by Bill Wichser
core_power_limit_count core_throttle_count package_power_limit_count
package_throttle_count
18781048
0
18781097
0
This was what led us to how the chassis was limiting power. We had been

I don't mean to be pedantic, but to me, this is the cpu throttling itself,
based on its own temperature readings and power rating. the coretemp
module, from its modinfo, seems to be purely on-chip.

/sys/bus/platform/devices/coretemp.0 probably contains some other
stuff which might be interesting - for instance, what your *_max
values are.

Post by Bill Wichser
using redundancy and switched to non-redundant to try and eliminate. We
believe that we see these messages when the CPU is throttling up in power.

I read the *_limit_count as meaning "18781048 times the core was
down-clocked because it exceeded power limits." ie, not "throttling up",
though I suppose these things are almost symmetric...
so spec is 115 W and Tcase max 80C. that's not as low a threshold
as some chips (67C seems pretty low, for instance).

regards, mark.

[Lux, Jim (337C)]

Jim Lux


-----Original Message-----
From: beowulf-***@beowulf.org [mailto:beowulf-***@beowulf.org] On Behalf Of Mark Hahn
Sent: Friday, August 30, 2013 10:48 AM
To: Bill Wichser
Cc: ***@beowulf.org
Subject: Re: [Beowulf] Problems with Dell M620 and CPU power throttling

Post by Bill Wichser
core_power_limit_count core_throttle_count package_power_limit_count
package_throttle_count
18781048
0
18781097
0
This was what led us to how the chassis was limiting power. We had been

I don't mean to be pedantic, but to me, this is the cpu throttling itself, based on its own temperature readings and power rating. the coretemp module, from its modinfo, seems to be purely on-chip.

[Lux, Jim (337C)]

Note that the time constant from on-chip sensor and on-chip throttle might be pretty short. You could see intermittent throttling that wouldn't manifest itself with external temperature measurements (or even reading the on-chip sensor) every 10 seconds.

Jim Lux


-----Original Message-----
From: beowulf-***@beowulf.org [mailto:beowulf-***@beowulf.org] On Behalf Of Mark Hahn
Sent: Friday, August 30, 2013 10:48 AM
To: Bill Wichser
Cc: ***@beowulf.org
Subject: Re: [Beowulf] Problems with Dell M620 and CPU power throttling

Post by Bill Wichser
core_power_limit_count core_throttle_count package_power_limit_count
package_throttle_count
18781048
0
18781097
0
This was what led us to how the chassis was limiting power. We had been

I don't mean to be pedantic, but to me, this is the cpu throttling itself, based on its own temperature readings and power rating. the coretemp module, from its modinfo, seems to be purely on-chip.

/sys/bus/platform/devices/coretemp.0 probably contains some other stuff which might be interesting - for instance, what your *_max values are.

Post by Bill Wichser
using redundancy and switched to non-redundant to try and eliminate.
We believe that we see these messages when the CPU is throttling up in power.

I read the *_limit_count as meaning "18781048 times the core was down-clocked because it exceeded power limits." ie, not "throttling up", though I suppose these things are almost symmetric...
so spec is 115 W and Tcase max 80C. that's not as low a threshold as some chips (67C seems pretty low, for instance).

regards, mark.
_______________________________________________
Beowulf mailing list, ***@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

[Lux, Jim (337C)]

Have you tried measuring the AC power with a Kill-A-Watt type device (or from your power distribution unit).. That's a lot easier than trying to measure the DC power going to the processor, and might tell you something useful, diagnostics wise. If the offending unit shows a different AC power consumption before hiccupping, for instance.

Jim Lux

-----Original Message-----
From: beowulf-***@beowulf.org [mailto:beowulf-***@beowulf.org] On Behalf Of Bill Wichser
Sent: Friday, August 30, 2013 9:20 AM
To: Mark Hahn
Cc: ***@beowulf.org
Subject: Re: [Beowulf] Problems with Dell M620 and CPU power throttling

Post by Bill Wichser
Of course we have done system tuning.

sorry for the unintentional condescenscion - what I actually meant was
"tuning of knobs located in /sys" :)

Post by Bill Wichser
Instrumenting temperature probes on individual CPUs has not been
performed. When we look at temperatures from both the chassis and
ipmitool, we see no drastic peaks. Maybe we are getting a 60C peak
that we don't detect and that is the cause. But I doubt it.

could you try "modprobe coretemp", and see whether interesting things
/sys/devices/system/cpu/cpu*/thermal_throttle/core_throttle_count
afaik, reading the coretemp*/temp*_input values would let you do
higher-resolution monitoring to see whether you're getting spikes.

We have these already loaded and see values:

[***@r2c3n4 thermal_throttle]# ls
core_power_limit_count core_throttle_count package_power_limit_count package_throttle_count
[***@r2c3n4 thermal_throttle]# cat *
18781048
0
18781097
0

This was what led us to how the chassis was limiting power. We had been using redundancy and switched to non-redundant to try and eliminate. We believe that we see these messages when the CPU is throttling up in power. From the google oracle, these messages are benign. Perhaps that isn't so...

Post by Bill Wichser
power consumption is around 80W. That tells me that the system is
cool enough. Should I not believe those values? i have no reason to
from past experience.

I'm not casting aspersions, just that chassis temps don't tell the
whole story. is your exact model of CPU actually rated for higher power?
we've got some ProLiant SL230s Gen8 with E5-2680's - rated for 130,
and don't seem to be throttling.

Post by Bill Wichser
Input air is about 22C. For our data center, you'd have a better
chance of getting this adjusted to 15C than I would! As for fans,
these don't have

yes, well, it is nice to have one's own datacenter ;) but seriously, I
find it sometimes makes a difference to open front and back doors of
the rack (if any), do some manual sampling of air flow and
temperatures (wave hand around)...

Post by Bill Wichser
For heat sink thermal grease problems, I'd expect this to be visible
using the ipmitools but maybe that is not where the temperatures are
being measured. I don't know about that issue. I'd expect that a
bad thermal grease issue would manifest itself by showing up on a per
socket level and not on both sockets. It seems odd that every node
exhibiting this problem would have both sockets having the same issue.

well, if both sockets have poor thermal contact with heatsinks...
I'm not trying to FUD up any particular vendor(s), but mistakes do happen.
I was imagining, for instance, that an assembly line might be set up
with HS and thermal compound tuned for E5-2637 systems (80W/socket),
but was pressed into service for some E5-2690 nodes (135W).

I'd expect to see the bad nodes be bad nodes consistently. They have been mostly moving targets at this point, randomly distributed.

Post by Bill Wichser
Again, the magnitude of the problem is about 5-10% at any time.
Given
600

if I understand you, the prevalence is only 5-10%, but the magnitude (effect)
is much larger, right?

Right. We have many jobs which use 32 nodes or more. Anytime a node goes bad, the whole job starts to crawl along thus tying up resources for days instead of hours.

Bill

regards, mark.

_______________________________________________
Beowulf mailing list, ***@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

[Bill Wichser]

Thanks for everyone's suggestions so far. I have a very good lead,
which we have tested, and appears to have promising effects. I will
summarize after we have tested on a few more nodes and confirmed some
results.

Thanks,
Bill

[Don Holmgren]

It might be worth fooling a bit with the cpufreq settings, down in

/sys/devices/system/cpu/cpuX/cpufreq

(where X=cpu#, one per core) To prevent non-thermal throttling, you can do
for each core

echo userspace > scaling_governor
cat scaling_max_freq
echo 2400000 > scaling_setspeed

(where substitute the max_freq reported for the 2400000). For this to work you
need the specific cpufreq driver for your processor loaded. For our (non-Dell)
SB servers it's acpi_cpufreq. In RedHat, the cpuspeed service loads the
relevent drivers, not sure if there is a similar service in other distros.

The above will lock the the cores at the max_freq, although if they get too hot
they will still throttle down in speed. There are statistics available on
frequency changes from thermal throttling in

/sys/devices/system/cpu/cpu0/thermal_throttle/

although I haven't used them, so I'm not sure about their functionality.

If you do a

modprobe cpufreq_stats

then a new directory

/sys/devices/system/cpu/cpu0/cpufreq/stats

will show up that has statistics about cpu speed changes. I'm not sure whether
thermal throttling changes will also show here or not. On one of our large
Opteron clusters, we had a handful of nodes with somewhat similar slowdown
problems as you are seeing on your SB's. We now lock their frequencies, and we
monitor /sys/devices/system/cpu/cpu0/cpufreq/stats/total_trans (which give total
number of speed changes), alarming when total_trans is non-zero.

Don Holmgren
Fermilab

Post by Bill Wichser
Since January, when we installed an M620 Sandybridge cluster from Dell,
we have had issues with power and performance to compute nodes. Dell
apparently continues to look into the problem but the usual responses
have provided no solution. Firmware, BIOS, OS updates all are fruitless.
The problem is that the node/CPU is power capped. We first detected
this with the STREAM benchmark, a quick run, which shows memory
bandwidth around 2000 instead of the normal 13000 MB/s. When the CPU is
in the C0 state, this drops to around 600.
The effect appears randomly across the entire cluster with 5-10% of the
nodes demonstrating some slower performance. We don't know what
triggers this. Using "turbostat" we can see that the GHz of the cores
is >= 1 in most cases, dropping to about 0.2 in some of the worst cases.
Looking at the power consumption by either the chassis GUI or using
"impitool sdr list" we see that there is only about 80 watts being used.
We run the RH 6.x release and are up to date with kernel/OS patches.
All firmware is up to date. Chassis power is configured as
non-redundant. tuned is set for performance. Turbo mode is
on/hyperthreading is off/performance mode is set in BIOS.
A reboot does not change this problem. But a power cycle returns the
compute node to normal again. Again, we do not know what triggers this
event. We are not overheating the nodes. But while applications are
running, something triggers an event where this power capping takes effect.
At this point we remain clueless about what is causing this to happen.
We can detect the condition now and have been power cycling the nodes in
order to reset.
If anyone has a clue, or better yet, solved the issue, we'd love to hear
the solution!
Thanks,
Bill

[Bill Wichser]

The solution appears to be BIOS configuration.

We had:
< SysProfile=perfoptimized
< ;ProcPwrPerf=maxperf
< ;ProcC1E=disable
< ;ProcCStates=disable

And changed to:
---

SysProfile=custom
ProcPwrPerf=osdbpm
ProcC1E=enable
ProcCStates=enable

Then added
modprobe acpi_cpufreq

And we move from BIOS directed power control to OS enabled power
control. While in the old mode we could set the processor states but
were unable to see some of the hooks Don had suggested here.

Initial results look good. We have a much better view of what the cores
are actually doing using the cpupower command, info we were unable to
obtain completely without this BIOS change.

I'm not sure about the C1E state being enabled though and will
experiment further.

Thanks to everyone who offered suggestions. An extra thanks to Don
Holmgren who pointed us down this path.

Bill

It might be worth fooling a bit with the cpufreq settings, down in
/sys/devices/system/cpu/cpuX/cpufreq
(where X=cpu#, one per core) To prevent non-thermal throttling, you can do
for each core
echo userspace > scaling_governor
cat scaling_max_freq
echo 2400000 > scaling_setspeed
(where substitute the max_freq reported for the 2400000). For this to
work you need the specific cpufreq driver for your processor loaded.
For our (non-Dell) SB servers it's acpi_cpufreq. In RedHat, the
cpuspeed service loads the relevent drivers, not sure if there is a
similar service in other distros.
The above will lock the the cores at the max_freq, although if they get
too hot they will still throttle down in speed. There are statistics
available on frequency changes from thermal throttling in
/sys/devices/system/cpu/cpu0/thermal_throttle/
although I haven't used them, so I'm not sure about their functionality.
If you do a
modprobe cpufreq_stats
then a new directory
/sys/devices/system/cpu/cpu0/cpufreq/stats
will show up that has statistics about cpu speed changes. I'm not sure
whether thermal throttling changes will also show here or not. On one
of our large Opteron clusters, we had a handful of nodes with somewhat
similar slowdown problems as you are seeing on your SB's. We now lock
their frequencies, and we monitor
/sys/devices/system/cpu/cpu0/cpufreq/stats/total_trans (which give total
number of speed changes), alarming when total_trans is non-zero.
Don Holmgren
Fermilab

Post by Bill Wichser
Since January, when we installed an M620 Sandybridge cluster from Dell,
we have had issues with power and performance to compute nodes. Dell
apparently continues to look into the problem but the usual responses
have provided no solution. Firmware, BIOS, OS updates all are fruitless.
The problem is that the node/CPU is power capped. We first detected
this with the STREAM benchmark, a quick run, which shows memory
bandwidth around 2000 instead of the normal 13000 MB/s. When the CPU is
in the C0 state, this drops to around 600.
The effect appears randomly across the entire cluster with 5-10% of the
nodes demonstrating some slower performance. We don't know what
triggers this. Using "turbostat" we can see that the GHz of the cores
is >= 1 in most cases, dropping to about 0.2 in some of the worst cases.
Looking at the power consumption by either the chassis GUI or using
"impitool sdr list" we see that there is only about 80 watts being used.
We run the RH 6.x release and are up to date with kernel/OS patches.
All firmware is up to date. Chassis power is configured as
non-redundant. tuned is set for performance. Turbo mode is
on/hyperthreading is off/performance mode is set in BIOS.
A reboot does not change this problem. But a power cycle returns the
compute node to normal again. Again, we do not know what triggers this
event. We are not overheating the nodes. But while applications are
running, something triggers an event where this power capping takes effect.
At this point we remain clueless about what is causing this to happen.
We can detect the condition now and have been power cycling the nodes in
order to reset.
If anyone has a clue, or better yet, solved the issue, we'd love to hear
the solution!
Thanks,
Bill

[Brice Goglin]

Hello,
I am seeing messages like this quite often on our R720s in dmesg:
CPU16: Core power limit notification (total events = 1)
Do you think that's related to your problem?
Brice

Post by Bill Wichser
The solution appears to be BIOS configuration.
< SysProfile=perfoptimized
< ;ProcPwrPerf=maxperf
< ;ProcC1E=disable
< ;ProcCStates=disable
---

SysProfile=custom
ProcPwrPerf=osdbpm
ProcC1E=enable
ProcCStates=enable

Then added
modprobe acpi_cpufreq
And we move from BIOS directed power control to OS enabled power
control. While in the old mode we could set the processor states but
were unable to see some of the hooks Don had suggested here.
Initial results look good. We have a much better view of what the cores
are actually doing using the cpupower command, info we were unable to
obtain completely without this BIOS change.
I'm not sure about the C1E state being enabled though and will
experiment further.
Thanks to everyone who offered suggestions. An extra thanks to Don
Holmgren who pointed us down this path.
Bill

It might be worth fooling a bit with the cpufreq settings, down in
/sys/devices/system/cpu/cpuX/cpufreq
(where X=cpu#, one per core) To prevent non-thermal throttling, you can do
for each core
echo userspace > scaling_governor
cat scaling_max_freq
echo 2400000 > scaling_setspeed
(where substitute the max_freq reported for the 2400000). For this to
work you need the specific cpufreq driver for your processor loaded.
For our (non-Dell) SB servers it's acpi_cpufreq. In RedHat, the
cpuspeed service loads the relevent drivers, not sure if there is a
similar service in other distros.
The above will lock the the cores at the max_freq, although if they get
too hot they will still throttle down in speed. There are statistics
available on frequency changes from thermal throttling in
/sys/devices/system/cpu/cpu0/thermal_throttle/
although I haven't used them, so I'm not sure about their functionality.
If you do a
modprobe cpufreq_stats
then a new directory
/sys/devices/system/cpu/cpu0/cpufreq/stats
will show up that has statistics about cpu speed changes. I'm not sure
whether thermal throttling changes will also show here or not. On one
of our large Opteron clusters, we had a handful of nodes with somewhat
similar slowdown problems as you are seeing on your SB's. We now lock
their frequencies, and we monitor
/sys/devices/system/cpu/cpu0/cpufreq/stats/total_trans (which give total
number of speed changes), alarming when total_trans is non-zero.
Don Holmgren
Fermilab

Post by Bill Wichser
Since January, when we installed an M620 Sandybridge cluster from Dell,
we have had issues with power and performance to compute nodes. Dell
apparently continues to look into the problem but the usual responses
have provided no solution. Firmware, BIOS, OS updates all are fruitless.
The problem is that the node/CPU is power capped. We first detected
this with the STREAM benchmark, a quick run, which shows memory
bandwidth around 2000 instead of the normal 13000 MB/s. When the CPU is
in the C0 state, this drops to around 600.
The effect appears randomly across the entire cluster with 5-10% of the
nodes demonstrating some slower performance. We don't know what
triggers this. Using "turbostat" we can see that the GHz of the cores
is >= 1 in most cases, dropping to about 0.2 in some of the worst cases.
Looking at the power consumption by either the chassis GUI or using
"impitool sdr list" we see that there is only about 80 watts being used.
We run the RH 6.x release and are up to date with kernel/OS patches.
All firmware is up to date. Chassis power is configured as
non-redundant. tuned is set for performance. Turbo mode is
on/hyperthreading is off/performance mode is set in BIOS.
A reboot does not change this problem. But a power cycle returns the
compute node to normal again. Again, we do not know what triggers this
event. We are not overheating the nodes. But while applications are
running, something triggers an event where this power capping takes effect.
At this point we remain clueless about what is causing this to happen.
We can detect the condition now and have been power cycling the nodes in
order to reset.
If anyone has a clue, or better yet, solved the issue, we'd love to hear
the solution!
Thanks,
Bill

_______________________________________________
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

[Bill Wichser]

Yes. Especially when you do not see the power limit normal following
it. This says that the cores are limited (by power) but never actually
receiving enough to then say that they are normal again.

We'd see both on ramp up to C0 state. limited then normal; limited then
normal.

Bill

Post by Brice Goglin
Hello,
CPU16: Core power limit notification (total events = 1)
Do you think that's related to your problem?
Brice

Post by Bill Wichser
The solution appears to be BIOS configuration.
< SysProfile=perfoptimized
< ;ProcPwrPerf=maxperf
< ;ProcC1E=disable
< ;ProcCStates=disable
---

SysProfile=custom
ProcPwrPerf=osdbpm
ProcC1E=enable
ProcCStates=enable

Then added
modprobe acpi_cpufreq
And we move from BIOS directed power control to OS enabled power
control. While in the old mode we could set the processor states but
were unable to see some of the hooks Don had suggested here.
Initial results look good. We have a much better view of what the cores
are actually doing using the cpupower command, info we were unable to
obtain completely without this BIOS change.
I'm not sure about the C1E state being enabled though and will
experiment further.
Thanks to everyone who offered suggestions. An extra thanks to Don
Holmgren who pointed us down this path.
Bill

It might be worth fooling a bit with the cpufreq settings, down in
/sys/devices/system/cpu/cpuX/cpufreq
(where X=cpu#, one per core) To prevent non-thermal throttling, you can do
for each core
echo userspace > scaling_governor
cat scaling_max_freq
echo 2400000 > scaling_setspeed
(where substitute the max_freq reported for the 2400000). For this to
work you need the specific cpufreq driver for your processor loaded.
For our (non-Dell) SB servers it's acpi_cpufreq. In RedHat, the
cpuspeed service loads the relevent drivers, not sure if there is a
similar service in other distros.
The above will lock the the cores at the max_freq, although if they get
too hot they will still throttle down in speed. There are statistics
available on frequency changes from thermal throttling in
/sys/devices/system/cpu/cpu0/thermal_throttle/
although I haven't used them, so I'm not sure about their functionality.
If you do a
modprobe cpufreq_stats
then a new directory
/sys/devices/system/cpu/cpu0/cpufreq/stats
will show up that has statistics about cpu speed changes. I'm not sure
whether thermal throttling changes will also show here or not. On one
of our large Opteron clusters, we had a handful of nodes with somewhat
similar slowdown problems as you are seeing on your SB's. We now lock
their frequencies, and we monitor
/sys/devices/system/cpu/cpu0/cpufreq/stats/total_trans (which give total
number of speed changes), alarming when total_trans is non-zero.
Don Holmgren
Fermilab

Post by Bill Wichser
Since January, when we installed an M620 Sandybridge cluster from Dell,
we have had issues with power and performance to compute nodes. Dell
apparently continues to look into the problem but the usual responses
have provided no solution. Firmware, BIOS, OS updates all are fruitless.
The problem is that the node/CPU is power capped. We first detected
this with the STREAM benchmark, a quick run, which shows memory
bandwidth around 2000 instead of the normal 13000 MB/s. When the CPU is
in the C0 state, this drops to around 600.
The effect appears randomly across the entire cluster with 5-10% of the
nodes demonstrating some slower performance. We don't know what
triggers this. Using "turbostat" we can see that the GHz of the cores
is >= 1 in most cases, dropping to about 0.2 in some of the worst cases.
Looking at the power consumption by either the chassis GUI or using
"impitool sdr list" we see that there is only about 80 watts being used.
We run the RH 6.x release and are up to date with kernel/OS patches.
All firmware is up to date. Chassis power is configured as
non-redundant. tuned is set for performance. Turbo mode is
on/hyperthreading is off/performance mode is set in BIOS.
A reboot does not change this problem. But a power cycle returns the
compute node to normal again. Again, we do not know what triggers this
event. We are not overheating the nodes. But while applications are
running, something triggers an event where this power capping takes effect.
At this point we remain clueless about what is causing this to happen.
We can detect the condition now and have been power cycling the nodes in
order to reset.
If anyone has a clue, or better yet, solved the issue, we'd love to hear
the solution!
Thanks,
Bill

_______________________________________________
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

[Christopher Samuel]

Post by Bill Wichser
Since January, when we installed an M620 Sandybridge cluster from
Dell, we have had issues with power and performance to compute
nodes. Dell apparently continues to look into the problem but the
usual responses have provided no solution. Firmware, BIOS, OS
updates all are fruitless.

One question, have you seen either the kernel or the BMC reporting
thermal throttling? For instance dmesg should show you something like:

CPU0: Core temperature above threshold, cpu clock throttled (total
events = 545939)
CPU0: Core temperature/speed normal

If you're not then there is one other possibility that you may like to
test, which is tell the kernel to not automatically turn on all
powersaving modes as that introduces a heap of latency (and
potentially other issues).

We pass through:

intel_idle.max_cstate=0 processor.max_cstate=1

on our SandyBridge nodes for just that reason.

If you don't then the kernel will say "Oh, this is an Intel CPU, I
know this!" (to paraphrase Jurassic Park) and enable every power
saving feature it can find, regardless of what your BIOS/UEFI is set to.

Best of luck!
Chris
- --
Christopher Samuel Senior Systems Administrator
VLSCI - Victorian Life Sciences Computation Initiative
Email: ***@unimelb.edu.au Phone: [phone]
http://www.vlsci.org.au/ http://twitter.com/vlsci

[Joe Landman]

Post by Christopher Samuel
intel_idle.max_cstate=0 processor.max_cstate=1
on our SandyBridge nodes for just that reason.

Yes. Mandatory on Sandy Bridge. We often include cpu.idle=0 to handle
an occasionally observed idle bug.

This is reminiscent of the "pausing" bug in Nehalem. C state power
transition, and suddenly the IOH/CPU loses seconds worth of clock ticks.
Very annoying.

Post by Christopher Samuel
If you don't then the kernel will say "Oh, this is an Intel CPU, I
know this!" (to paraphrase Jurassic Park) and enable every power
saving feature it can find, regardless of what your BIOS/UEFI is set to.

:)

Power savings is great, when it works. When it doesn't work, the
wreckage thats left is epic.
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: ***@scalableinformatics.com
web : http://scalableinformatics.com
http://scalableinformatics.com/siflash
phone: [phone] x121
fax : [phone]
cell : [phone]

[Bill Wichser]

Sigh. I'm back with the same issues.

We moved the power management entirely into the OS by changing BIOS and
chassis settings. We now can see more info using the OS acpi commands.
This isn't a state issue. We added the kernel parameters
intel_idle.max_cstate=0 processor.max_cstate=1 to eliminate the Intel
throttles turning it over to the OS. Still same issues.

turbostat shows core speeds at >1.2 GHz. i7z shows that temps are below
60C. When the cores drift below the 2.6GHz we have, we send values with
cpupower command or by trying to directly set using
/sys/devices/system/cpu/cpuX/cpufreq and while the change is taken, no
actual change ever takes place.

cpupower frequency-info

shows, initially,
current policy: frequency should be within 1.20 GHz and 2.60 GHz
current CPU frequency is 2.60 GHz (asserted by call to hardware).

starting an HPL run immediately takes this to
current policy: frequency should be within 1.20 GHz and 2.20 GHz.
current CPU frequency is 1.60 GHz (asserted by call to hardware).

This is a "good" node. Others drop to about 0.2GHz. Again i7z shows
temps in the mid 50C range.

ipmitool sdr shows
Pwr Consumption | 192 Watts | ok
Current | 0.80 Amps | ok


On a normal node the power is upwards of 350W.

We are trying to escalate with Dell but that process is SLOW!

Thanks,
Bill

Post by Bill Wichser
Since January, when we installed an M620 Sandybridge cluster from Dell,
we have had issues with power and performance to compute nodes. Dell
apparently continues to look into the problem but the usual responses
have provided no solution. Firmware, BIOS, OS updates all are fruitless.
The problem is that the node/CPU is power capped. We first detected
this with the STREAM benchmark, a quick run, which shows memory
bandwidth around 2000 instead of the normal 13000 MB/s. When the CPU is
in the C0 state, this drops to around 600.
The effect appears randomly across the entire cluster with 5-10% of the
nodes demonstrating some slower performance. We don't know what
triggers this. Using "turbostat" we can see that the GHz of the cores
is >= 1 in most cases, dropping to about 0.2 in some of the worst cases.
Looking at the power consumption by either the chassis GUI or using
"impitool sdr list" we see that there is only about 80 watts being used.
We run the RH 6.x release and are up to date with kernel/OS patches.
All firmware is up to date. Chassis power is configured as
non-redundant. tuned is set for performance. Turbo mode is
on/hyperthreading is off/performance mode is set in BIOS.
A reboot does not change this problem. But a power cycle returns the
compute node to normal again. Again, we do not know what triggers this
event. We are not overheating the nodes. But while applications are
running, something triggers an event where this power capping takes effect.
At this point we remain clueless about what is causing this to happen.
We can detect the condition now and have been power cycling the nodes in
order to reset.
If anyone has a clue, or better yet, solved the issue, we'd love to hear
the solution!
Thanks,
Bill
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

[Richard Hickey]

Post by Bill Wichser
Since January, when we installed an M620 Sandybridge cluster from Dell,
we have had issues with power and performance to compute nodes. Dell
apparently continues to look into the problem but the usual responses
have provided no solution. Firmware, BIOS, OS updates all are fruitless.
The problem is that the node/CPU is power capped. We first detected
this with the STREAM benchmark, a quick run, which shows memory
bandwidth around 2000 instead of the normal 13000 MB/s. When the CPU is
in the C0 state, this drops to around 600.
The effect appears randomly across the entire cluster with 5-10% of the
nodes demonstrating some slower performance. We don't know what
triggers this. Using "turbostat" we can see that the GHz of the cores
is >= 1 in most cases, dropping to about 0.2 in some of the worst cases.
Looking at the power consumption by either the chassis GUI or using
"impitool sdr list" we see that there is only about 80 watts being used.
We run the RH 6.x release and are up to date with kernel/OS patches.
All firmware is up to date. Chassis power is configured as
non-redundant. tuned is set for performance. Turbo mode is
on/hyperthreading is off/performance mode is set in BIOS.
A reboot does not change this problem. But a power cycle returns the
compute node to normal again. Again, we do not know what triggers this
event. We are not overheating the nodes. But while applications are
running, something triggers an event where this power capping takes effect.
At this point we remain clueless about what is causing this to happen.
We can detect the condition now and have been power cycling the nodes in
order to reset.
If anyone has a clue, or better yet, solved the issue, we'd love to hear
the solution!
Thanks,
Bill

You are not alone in seeing this. We discovered it via by some of our
weather codes running slow. A co-worker started running single node Linpack
runs and we saw individual nodes running slow. A reboot did not fix,
however a power cycle did. We can see a 2 to 3 fold increase in
performance.



We found that you could either do a physical reseat of the blade, or a
logical one through the cmc command line. Either way fixes the problem
temporarily.



It's good to see that someone else is seeing this. Well, maybe not good,
but at least we're not the only ones fighting this.





Rich

[Bill Wichser]

One week ago, on Tuesday September 10, we had a system downtime for some
work on our central filesystem. During that outage we decided to not
only power of the nodes but to also power off the chassis across 10
racks of these blades.

Every day we check the systems by running a "stress" benchmark and
checking the CPU speeds with "turbostat" and every day we find that the
cores are all exceeding the 2.6GHz E5-2670 rating, mostly settling in at
3.0 GHz (with turbo mode enabled). Every day I wait to find that things
have again degraded but have been happily surprised.

If we get through a whole month then I would say that after all the
firmware and iDrac and CMC updates that a chassis power cycle is the
answer. But tomorrow I will look again and hopefully be happily
surprised for one more day.

Bill

Post by Richard Hickey
You are not alone in seeing this. We discovered it via by some of our
weather codes running slow. A co-worker started running single node Linpack
runs and we saw individual nodes running slow. A reboot did not fix,
however a power cycle did. We can see a 2 to 3 fold increase in
performance.
We found that you could either do a physical reseat of the blade, or a
logical one through the cmc command line. Either way fixes the problem
temporarily.
It's good to see that someone else is seeing this. Well, maybe not good,
but at least we're not the only ones fighting this.
Rich
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

[Christopher Samuel]

Post by Bill Wichser
If we get through a whole month then I would say that after all the
firmware and iDrac and CMC updates that a chassis power cycle is
the answer.

One possibility there is that there were some firmware updates that
only took after you power-cycled the chassis.

Great to hear it's all sorted now!

All the best,
Chris
- --
Christopher Samuel Senior Systems Administrator
VLSCI - Victorian Life Sciences Computation Initiative
Email: ***@unimelb.edu.au Phone: [phone]
http://www.vlsci.org.au/ http://twitter.com/vlsci

[Douglas O'Flaherty]

I recently received a hallway tip from a performance engineer at a major SW company:

"Run in C1. C0 over commits unpredictably, then throttles."

He didn't specify a hardware platform.


AFK: mobile

Post by Bill Wichser

Post by Bill Wichser
Since January, when we installed an M620 Sandybridge cluster from Dell,
we have had issues with power and performance to compute nodes. Dell
apparently continues to look into the problem but the usual responses
have provided no solution. Firmware, BIOS, OS updates all are fruitless.
The problem is that the node/CPU is power capped. We first detected
this with the STREAM benchmark, a quick run, which shows memory
bandwidth around 2000 instead of the normal 13000 MB/s. When the CPU is
in the C0 state, this drops to around 600.
The effect appears randomly across the entire cluster with 5-10% of the
nodes demonstrating some slower performance. We don't know what
triggers this. Using "turbostat" we can see that the GHz of the cores
is >= 1 in most cases, dropping to about 0.2 in some of the worst cases.
Looking at the power consumption by either the chassis GUI or using
"impitool sdr list" we see that there is only about 80 watts being used.
We run the RH 6.x release and are up to date with kernel/OS patches.
All firmware is up to date. Chassis power is configured as
non-redundant. tuned is set for performance. Turbo mode is
on/hyperthreading is off/performance mode is set in BIOS.
A reboot does not change this problem. But a power cycle returns the
compute node to normal again. Again, we do not know what triggers this
event. We are not overheating the nodes. But while applications are
running, something triggers an event where this power capping takes

effect.

Post by Bill Wichser
At this point we remain clueless about what is causing this to happen.
We can detect the condition now and have been power cycling the nodes in
order to reset.
If anyone has a clue, or better yet, solved the issue, we'd love to hear
the solution!
Thanks,
Bill

You are not alone in seeing this. We discovered it via by some of our
weather codes running slow. A co-worker started running single node Linpack
runs and we saw individual nodes running slow. A reboot did not fix,
however a power cycle did. We can see a 2 to 3 fold increase in
performance.
We found that you could either do a physical reseat of the blade, or a
logical one through the cmc command line. Either way fixes the problem
temporarily.
It's good to see that someone else is seeing this. Well, maybe not good,
but at least we're not the only ones fighting this.
Rich
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

[Christopher Samuel]

Post by Douglas O'Flaherty
"Run in C1. C0 over commits unpredictably, then throttles."

I've seen a recommendation in a public Mellanox document of using C1
not C0 when using hyperthreading/SMT, could be related to this..

- --
Christopher Samuel Senior Systems Administrator
VLSCI - Victorian Life Sciences Computation Initiative
Email: ***@unimelb.edu.au Phone: [phone]
http://www.vlsci.org.au/ http://twitter.com/vlsci

[Bill Wichser]

We have tested using c1 instead of c0 but no difference. We don't use
logical processors at all. When the problems happens, it doesn't matter
what you set the cores for C1/C0, they never get up to speed again
without a power cycle/reseat. We believe this to be something related
to power. Maybe current limiting.

As I stated yesterday, after a complete chassis power cycle on Tuesday
Sept 10, the entire 37 chassis have been outperforming their 2.6GHz
ratings flawlessly. I don't know if this is going to be the solution we
have been searching to find but it has certainly been a week and a half
of some very happy researchers!

Thanks,
Bill

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Douglas O'Flaherty
"Run in C1. C0 over commits unpredictably, then throttles."

I've seen a recommendation in a public Mellanox document of using C1
not C0 when using hyperthreading/SMT, could be related to this..
- --
Christopher Samuel Senior Systems Administrator
VLSCI - Victorian Life Sciences Computation Initiative
http://www.vlsci.org.au/ http://twitter.com/vlsci
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlI7GPAACgkQO2KABBYQAh9zPQCfeOCdUupjqx7SDeFxQjBWG9NU
FL4AnRYA3zLCNzEVNp0ypiW9KMYp3ohW
=ntfO
-----END PGP SIGNATURE-----
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

[Aaron Knister]

Post by Bill Wichser
We have tested using c1 instead of c0 but no difference. We don't use
logical processors at all. When the problems happens, it doesn't matter
what you set the cores for C1/C0, they never get up to speed again
without a power cycle/reseat. We believe this to be something related
to power. Maybe current limiting.
As I stated yesterday, after a complete chassis power cycle on Tuesday
Sept 10, the entire 37 chassis have been outperforming their 2.6GHz
ratings flawlessly. I don't know if this is going to be the solution we
have been searching to find but it has certainly been a week and a half
of some very happy researchers!
Thanks,
Bill

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Douglas O'Flaherty
"Run in C1. C0 over commits unpredictably, then throttles."

I've seen a recommendation in a public Mellanox document of using C1
not C0 when using hyperthreading/SMT, could be related to this..
- --
Christopher Samuel Senior Systems Administrator
VLSCI - Victorian Life Sciences Computation Initiative
Email: samuel <at> unimelb.edu.au Phone: [phone]
http://www.vlsci.org.au/ http://twitter.com/vlsci
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlI7GPAACgkQO2KABBYQAh9zPQCfeOCdUupjqx7SDeFxQjBWG9NU
FL4AnRYA3zLCNzEVNp0ypiW9KMYp3ohW
=ntfO
-----END PGP SIGNATURE-----
_______________________________________________
Beowulf mailing list, Beowulf <at> beowulf.org sponsored by Penguin

Computing

Post by Bill Wichser

To change your subscription (digest mode or unsubscribe) visit

http://www.beowulf.org/mailman/listinfo/beowulf
Hi Bill,

I'm wondering if this issue has resurfaced for you after the firmware
updates and chassis power cycles?

I'm having what sounds to be the same issue but with R320's. So far
BIOS/iDRAC/Lifecycle controller updates haven't helped but I haven't tried
physically removing power to the node. I have been doing using the "ipmitool
power cycle" command to reboot the nodes and get them out of their funk
(running at 0.2GHz) but that, of course, still leaves part of the chassis
energized.

Thanks!

-Aaron

[Bill Wichser]

Post by Lux, Jim (337C)

Post by Bill Wichser
We have tested using c1 instead of c0 but no difference. We don't use
logical processors at all. When the problems happens, it doesn't matter
what you set the cores for C1/C0, they never get up to speed again
without a power cycle/reseat. We believe this to be something related
to power. Maybe current limiting.
As I stated yesterday, after a complete chassis power cycle on Tuesday
Sept 10, the entire 37 chassis have been outperforming their 2.6GHz
ratings flawlessly. I don't know if this is going to be the solution we
have been searching to find but it has certainly been a week and a half
of some very happy researchers!
Thanks,
Bill

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Douglas O'Flaherty
"Run in C1. C0 over commits unpredictably, then throttles."

I've seen a recommendation in a public Mellanox document of using C1
not C0 when using hyperthreading/SMT, could be related to this..
- --
Christopher Samuel Senior Systems Administrator
VLSCI - Victorian Life Sciences Computation Initiative
Email: samuel <at> unimelb.edu.au Phone: [phone]
http://www.vlsci.org.au/ http://twitter.com/vlsci
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlI7GPAACgkQO2KABBYQAh9zPQCfeOCdUupjqx7SDeFxQjBWG9NU
FL4AnRYA3zLCNzEVNp0ypiW9KMYp3ohW
=ntfO
-----END PGP SIGNATURE-----
_______________________________________________
Beowulf mailing list, Beowulf <at> beowulf.org sponsored by Penguin

Computing

Post by Bill Wichser

To change your subscription (digest mode or unsubscribe) visit

http://www.beowulf.org/mailman/listinfo/beowulf
Hi Bill,
I'm wondering if this issue has resurfaced for you after the firmware
updates and chassis power cycles?
I'm having what sounds to be the same issue but with R320's. So far
BIOS/iDRAC/Lifecycle controller updates haven't helped but I haven't tried
physically removing power to the node. I have been doing using the "ipmitool
power cycle" command to reboot the nodes and get them out of their funk
(running at 0.2GHz) but that, of course, still leaves part of the chassis
energized.
Thanks!
-Aaron
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

Aaron,

The problem has not resurfaced after the updates and power cycling of
the chassis themselves. Just doing the nodes never did help as the
firmware in the chassis itself was the one which needed the power cycle.

Bill