PC GAMESS/Firefly-related discussion club



Learn how to ask questions correctly


Re^3: Firefly, mpich and OpenIB

Alex Granovsky
gran@classic.chem.msu.su


Hi all,


Indeed, OpenMPI's OpenIB component works to some degree
strange at least. Using the default PC GAMESS/Firefly's
settings, the performance of communications is very bad,
thus resulting in overall performance degradation (this
is not the case for other supported MPIs working over IB).

However, using the settings Jim has already reported, namely:

 $SYSTEM  mxbcst=-1  mpisnc=1 $END
 $mpi mxgsum=1048576 $end

this problem can be easily fixed. The first two options
are described in the existing Firefly's documentation.

The last one sets the size of buffer (in 8-byte words)
for MPI-Allreduce calls.

Regards,
Alex Granovsky


On Tue Jan 6 '09 6:24pm, Jim Kress wrote
----------------------------------------
>Go to the pcgamess home page and look at the command line options. You'll see a number of them that will help resolve your problems.  

>For example, the -t option allows you to automatically create scratch files that are unique for each process you wish to spawn.

>I find the following works:

># /opt/openmpi-1.2.8-gcc32/bin/mpirun -np 32 -hostfile
>eth32 /home/Alex/pcgamess -r -f  -p -i /home/Alex/jwk_input.inp
>-o /home/Alex/t8x_jwk_noib_self.out -ex /home/Alex  -t /tmp/ff.ompi/test

>You also need to be sure you make the 32 bit shared libraries for openMPI 1.2.8.

>And you need to include the following in your input file:

>$SYSTEM  mxbcst=-1  mpisnc=1 $END
>$mpi mxgsum=1048576 $end

>Alex Granovsky (and to a much lesser extent, I) spent last weekend figuring this out.

>Be sure to add the 32 bit version into mpi-selector as a separate choice from the 64 bit version.  In either case, the current version in ROCKS v5.1 is 1.2.6 so you'll be updating to 1.2.8.  To get the 32 bit version us CCFLAGS= -m32 etc.

>Also, be sure you get the hostfile right.  It is not the same as the one used in mpich.

>Jim
>
>
>On Tue Jan 6 '09 4:35am, Jonas Baltrusaitis wrote
>-------------------------------------------------
>>Jim, please do share what you will find out as I can;t run pcgamess using any mpich on Rocks 5.1, it starts a single job and not parallel
>>
>>
>>On Mon Jan 5 '09 4:17pm, Jim Kress wrote
>>----------------------------------------
>>>I'm having a number of problems running Firefly on my ROCKS 5.1 cluster using mpich 1.2.7. I tried OpenIB (InfiniBand) but I lack the 32 bit shared libraries and even after getting those, the IB performance is poor.

>>>Do you have any suggestions that can help resolve these problems?

>>>Thanks.

>>>Jim


[ Previous ] [ Next ] [ Index ]           Wed Jan 7 '09 6:04pm
[ Reply ] [ Edit ] [ Delete ]           This message read 1185 times