Discussion:
connect Mellanox IB FDR adapter to QLogic IB QDR switch?
(too old to reply)
Marko Kobal
2013-06-06 08:52:01 UTC
Permalink
Hi,

I have a cluster with QLogic IB QDR switches (QLogic 12200 (iDPx)
36-port QDR InfiniBand Switch), currently there are servers with QLogic
QME7342 QDR HCA adapters connected to those IB switches.

I've got a couple of Mellanox IB FDR adapters (MT27500 Family
[ConnectX-3]) and I wonder if they would play nicely with existing IB
network.

Has anybody got any experiences in mixing QDR and FDR IB, especially
connecting Mellanox FDR adapters to QLogic QDR IB switches?
--
Kind regards, Marko Kobal
Christopher Samuel
2013-06-07 00:12:14 UTC
Permalink
Post by Marko Kobal
I've got a couple of Mellanox IB FDR adapters (MT27500 Family
[ConnectX-3]) and I wonder if they would play nicely with existing
IB network.
I would expect them to work as I believe it's all meant to be
backwardly compatible and the FDR HCA's should downgrade to QDR to
talk to the switch. It's not something I've tried though.

I'm not quite as sure what would happen if you were to go the other
way and plug a QDR adapter into an FDR switch with all FDR HCAs.

cheers,
Chris
- --
Christopher Samuel Senior Systems Administrator
VLSCI - Victorian Life Sciences Computation Initiative
Email: ***@unimelb.edu.au Phone: +61 (0)3 903 55545
http://www.vlsci.org.au/ http://twitter.com/vlsci
Håkon Bugge
2013-06-07 05:58:34 UTC
Permalink
Post by Marko Kobal
Hi,
I have a cluster with QLogic IB QDR switches (QLogic 12200 (iDPx)
36-port QDR InfiniBand Switch), currently there are servers with QLogic
QME7342 QDR HCA adapters connected to those IB switches.
I've got a couple of Mellanox IB FDR adapters (MT27500 Family
[ConnectX-3]) and I wonder if they would play nicely with existing IB
network.
IBM OEM'ed QLogic switches at some point in time, and used Mellanox HCAs. My impression was that that road had it bumps. There are some ex QLogic people on this list, which can correct me here.

One thing for sure, you must make sure the Mellanox firmware in the HCAs are configured to run IBTA link initialization. I expect they run a Mellanox proprietary initialization by default.
Post by Marko Kobal
Has anybody got any experiences in mixing QDR and FDR IB, especially
connecting Mellanox FDR adapters to QLogic QDR IB switches?
As stated above. In addition, the Uni New Hampshire (www.iol.unh.edu) has a network interoperability lab, which in the past has done work in this area.


Thanks, Håkon
Post by Marko Kobal
--
Kind regards, Marko Kobal
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
Marko Kobal
2013-06-09 08:01:51 UTC
Permalink
Hi,
Post by Håkon Bugge
One thing for sure, you must make sure the Mellanox firmware in the HCAs are configured to run IBTA link initialization. I expect they run a Mellanox proprietary initialization by default.
OK, I've installed one adapter in one of the servers. Adapter is detected by the server and OS (centos6), latest ofed installed. IB diag tools shows link is down, adapter's leds are off.

Obviously I have to set IBTA link init. Any hint on how can I achive this? I can't find any relevant info about this in the user's manual and I have absolutely no experience with Mellanox adapters. Any help would be aprichieted, thanks!

Kind regards, Marko.
Marko Kobal
2013-06-13 13:44:01 UTC
Permalink
Hi,

Well, I noticed that the links did eventually came up ... however it
took several hours to do the link initialization:

cat /var/log/messages | grep mlx4_core

node_1:
Jun 6 17:42:08 node_1 kernel: mlx4_core 0000:10:00.0: PCI INT A -> GSI
30 (level, low) -> IRQ 30
Jun 7 04:35:33 node_1 kernel: mlx4_core 0000:10:00.0: mlx4_ib: Port 1
logical link is up

node_2:
Jun 12 16:05:12 node_2 kernel: mlx4_core 0000:10:00.0: PCI INT A -> GSI
30 (level, low) -> IRQ 30
Jun 12 20:50:24 node_2 kernel: mlx4_core 0000:10:00.0: mlx4_ib: Port 1
logical link is up

The connection is now actually working, however the extremely long time
to make the link initialization is really weird (and annoying when doing
server or openibd restart) ... Anyone seen this kind of behaviour on
Mellanox adapters?


Kind regards, Marko Kobal
Marko Kobal
2013-06-21 15:43:11 UTC
Permalink
Hi,

Update: we've upgraded QLogic switches firmware to the latest version
and now the connection is established in a matter of a few seconds. So,
the problem is resolved!


Kind regards, Marko Kobal
CTO, Arctur d.o.o.

Loading...