[Beowulf] mlx 10g ethernet

[Michael Di Domenico]

does anyone have any additional tuning tips beyond the random things
on the net for 10g ethernet tuning? i have two identical servers
(Quanta Q71L-4U) attached to a single cisco 10g switch, using
identical (Mellanox MNPH28B-XTC) cards. when i run mpi bandwidth
tests between the two cards, i top out at 200-300MB/sec using IMB
sendrecv. if i use lustre lnet selftests, i can crank the max to
500MB/sec, but i can't seem to crest this.

i've monkied around with the sysctl params and the cards are in x8
slots at 2.5GT/s. i think they should be capable of driving at
~900MB/sec, but i can't seem to get there. i know i'm missing
something...
_______________________________________________
Beowulf mailing list, ***@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/ma

[Alex Chekholko]

The ESNet pages are pretty good:
https://fasterdata.es.net/host-tuning/
https://fasterdata.es.net/network-tuning/

Since you're on a local low-latency link, all the stuff about
high-bandwidth high-latency links doesn't apply to you, but the host tuning
parameters at least give you somewhere to look.

I recommend iperf for testing before IMB, make sure you can get line rate
throughput with an iperf test.

Post by Michael Di Domenico
does anyone have any additional tuning tips beyond the random things
on the net for 10g ethernet tuning? i have two identical servers
(Quanta Q71L-4U) attached to a single cisco 10g switch, using
identical (Mellanox MNPH28B-XTC) cards. when i run mpi bandwidth
tests between the two cards, i top out at 200-300MB/sec using IMB
sendrecv. if i use lustre lnet selftests, i can crank the max to
500MB/sec, but i can't seem to crest this.
i've monkied around with the sysctl params and the cards are in x8
slots at 2.5GT/s. i think they should be capable of driving at
~900MB/sec, but i can't seem to get there. i know i'm missing
something...
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf

[Michael Di Domenico]

thanks, most of that i've already tried. i don't have iperf, but i
did run qperf, which shows 1.0GB/sec and 16us latency between the two
hosts. but i'm not entirely sure i believe it

the only thing that made a difference so far is changing the rx tx
rings up 8192 on the host. i went from 200-300MB/sec to 300-400MB/sec
consistently. but i'm still far short

On Fri, Aug 4, 2017 at 12:47 PM, Alex Chekholko

Post by Alex Chekholko
https://fasterdata.es.net/host-tuning/
https://fasterdata.es.net/network-tuning/
Since you're on a local low-latency link, all the stuff about high-bandwidth
high-latency links doesn't apply to you, but the host tuning parameters at
least give you somewhere to look.
I recommend iperf for testing before IMB, make sure you can get line rate
throughput with an iperf test.

Post by Michael Di Domenico
does anyone have any additional tuning tips beyond the random things
on the net for 10g ethernet tuning? i have two identical servers
(Quanta Q71L-4U) attached to a single cisco 10g switch, using
identical (Mellanox MNPH28B-XTC) cards. when i run mpi bandwidth
tests between the two cards, i top out at 200-300MB/sec using IMB
sendrecv. if i use lustre lnet selftests, i can crank the max to
500MB/sec, but i can't seem to crest this.
i've monkied around with the sysctl params and the cards are in x8
slots at 2.5GT/s. i think they should be capable of driving at
~900MB/sec, but i can't seem to get there. i know i'm missing
something...
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf

_______________________________________________
Beowulf mailing list, ***@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listin

[Michael Di Domenico]

found a copy of iperf on my system, so i ran it. show 0-10sec
10.6GBytes/sec 9.08Gbit/sec

so given that qperf seems to agree with iperf, i guess it's an
interesting question now why, lustre lnet_selftest and IMB sendrecv
seem throttled at 500MB/sec

On Fri, Aug 4, 2017 at 1:21 PM, Michael Di Domenico

Post by Michael Di Domenico
thanks, most of that i've already tried. i don't have iperf, but i
did run qperf, which shows 1.0GB/sec and 16us latency between the two
hosts. but i'm not entirely sure i believe it
the only thing that made a difference so far is changing the rx tx
rings up 8192 on the host. i went from 200-300MB/sec to 300-400MB/sec
consistently. but i'm still far short
On Fri, Aug 4, 2017 at 12:47 PM, Alex Chekholko

Post by Alex Chekholko
https://fasterdata.es.net/host-tuning/
https://fasterdata.es.net/network-tuning/
Since you're on a local low-latency link, all the stuff about high-bandwidth
high-latency links doesn't apply to you, but the host tuning parameters at
least give you somewhere to look.
I recommend iperf for testing before IMB, make sure you can get line rate
throughput with an iperf test.

Post by Michael Di Domenico
does anyone have any additional tuning tips beyond the random things
on the net for 10g ethernet tuning? i have two identical servers
(Quanta Q71L-4U) attached to a single cisco 10g switch, using
identical (Mellanox MNPH28B-XTC) cards. when i run mpi bandwidth
tests between the two cards, i top out at 200-300MB/sec using IMB
sendrecv. if i use lustre lnet selftests, i can crank the max to
500MB/sec, but i can't seem to crest this.
i've monkied around with the sysctl params and the cards are in x8
slots at 2.5GT/s. i think they should be capable of driving at
~900MB/sec, but i can't seem to get there. i know i'm missing
something...
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf

_______________________________________________
Beowulf mailing list, ***@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) v

[Christopher Samuel]

Post by Michael Di Domenico
so given that qperf seems to agree with iperf, i guess it's an
interesting question now why, lustre lnet_selftest and IMB sendrecv
seem throttled at 500MB/sec

Is this over TCP/IP or using RoCE (RDMA over Converged Ethernet) ?

[Michael Di Domenico]

Just TCP/IP

On Sun, Aug 6, 2017 at 10:07 PM, Christopher Samuel

Post by Christopher Samuel

Post by Michael Di Domenico
so given that qperf seems to agree with iperf, i guess it's an
interesting question now why, lustre lnet_selftest and IMB sendrecv
seem throttled at 500MB/sec

Is this over TCP/IP or using RoCE (RDMA over Converged Ethernet) ?
--
Christopher Samuel Senior Systems Administrator
Melbourne Bioinformatics - The University of Melbourne
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

_______________________________________________
Beowulf mailing list, ***@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listi