[Beowulf] OpenMPI & Slurm: mpiexec/mpirun vs. srun

[Prentice Bisbal]

Beowulfers,

[Note: this was also posted to the OpenMPI mailing list. Posting here,
too, to get some 'neutral' responses ]

We use OpenMPI with Slurm as our scheduler, and a user has asked me
this: should they use mpiexec/mpirun or srun to start their MPI jobs
through Slurm?

My inclination is to use mpiexec, since that is the only method that's
(somewhat) defined in the MPI standard and therefore the most portable,
and the examples in the OpenMPI FAQ use mpirun. However, the Slurm
documentation on the schedmd website say to use srun with the --mpi=pmi
option. (See links below)

What are the pros/cons of using these two methods, other than the
portability issue I already mentioned? Does srun+pmi use a different
method to wire up the connections? Some things I read online seem to
indicate that. If slurm was built with PMI support, and OpenMPI was
built with Slurm support, does it really make any difference?

https://www.open-mpi.org/faq/?category=slurm
https://slurm.schedmd.com/mpi_guide.html#open_mpi

[Christopher Samuel]

Post by Prentice Bisbal
What are the pros/cons of using these two methods, other than the
portability issue I already mentioned? Does srun+pmi use a different
method to wire up the connections? Some things I read online seem to
indicate that. If slurm was built with PMI support, and OpenMPI was
built with Slurm support, does it really make any difference?

Benchmark. In (much) older versions of Slurm we would find that NAMD
built with OpenMPI would scale better with mpirun, but in more recent
versions (from 15.x onwards I believe) we found srun scaled better instead.

Whether that's because of differences in wireup or other issues I'm not
sure, but currently I recommend people use srun instead. You will also
get better accounting information.

With mpirun OMPI will start orted on the each node (via srun) and then
that will launch the MPI ranks. With srun slurmd will launch the MPI
ranks itself.

Hope this helps!
Chris