Firefly and PC GAMESS-related discussion club

Learn how to ask questions correctly

Alex Granovsky
gran@classic.chem.msu.su

this is evidently Intel MPI/DAPL issue rather than Firefly's problem.
First check whether you can run Firefly with Intel MPI using shared
memory/sockets (ssm) device, then check rdssm device. In the case you
still cannot run Firefly, check whether you can run MPI "hello world"
program with Intel MPI/rdssm. If it fails, this most likely means
that version of Intel MPI installed on this cluster is not compatible
with DAPL libraries and, more generally, OFED stack. Alternatively,
IPoIB may be mis-configured or not enabled at all.

You may try Intel MPI 4.0 as well.

Regards,
Alex Granovsky


On Mon Jun 6 '11 5:44pm, Alexey Terent'ev wrote
-----------------------------------------------
>Hi,
>I try to get started  the Firefly in the cluster in the parallel mode.
>But I have a problem.

>Input file BAopt.inp
> ! Minimize (Energy/Geometry) B3LYP
> $CONTRL COORD=CART ICHARG=0 MULT=1 RUNTYP=OPTIMIZE SCFTYP=RHF $END
> $DFT DFTTYP=B3LYP METHOD=GRID $END
> $p2p p2p=.t. dlb=.t. xdlb=.t. $end
> $BASIS DIFFSP=.true. GBASIS=N31 NGAUSS=6 POLAR=POPLE $END
> $SYSTEM MEMORY=150000 $END
> $DATA
> 6june11
> C1
> O 8.0 -0.0000000000 0.0000000000 0.0000000000
> H 1.0 -0.8000000000 0.8000000000 0.0000000000
> H 1.0  0.8000000000 0.8000000000 0.0000000000
> $END

>Calculation using the single process:
> #!/bin/bash
> #PBS -N BAopt.inp
> #PBS -l nodes=1:ppn=1
> #PBS -l walltime=00:03:00
> #PBS -j oe
> #PBS -V
> mpi_dir=/opt/intel/mpi-rt/3.2.2/bin/
> cd $PBS_O_WORKDIR
> PATH=$PBS_O_PATH
> export I_MPI_DEVICE=rdma
> export I_MPI_DEBUG=0
> export I_MPI_FALLBACK_DEVICE=disable
> export I_MPI_PIN_MODE=lib
> export I_MPI_PIN_PROCS=0,2,1,5,3,4,6,7
> mpirun -r ssh -machinefile $PBS_NODEFILE -np `cat $PBS_NODEFILE|wc -l` ./firefly -r -f -p -stdext -ex ~/ex/ -i ~/BAopt.inp -o ~/BAopt.out -t ~/tmp/

>Output file BAopt.out:
> Core i7 / Linux Firefly version running under Linux.
> Running on Intel CPU: Brand ID 0, Family 6, Model 26, Stepping 5
> CPU Brand String: Intel(R) Xeon(R) CPU X5560 @ 2.80GHz
> CPU Features : CMOV, MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, HTT, MWAIT, EM64T
> Data cache size: L1 32 KB, L2 256 KB, L3 8192 KB
> max # of cores/package : 8
> max # of threads/package : 16
> max cache sharing level : 16
> actual # of cores/package : 4
> actual # of threads/package : 8
> actual # of threads/core: 2
> Operating System successfully passed SSE support test.
> PARALLEL VERSION (INTEL MPI) RUNNING IN SERIAL MODE USING SINGLE PROCESS
> …
> CPU TIME: STEP = 0.00 , TOTAL = 0.2 SECONDS (0.0 MIN)
> WALL CLOCK TIME: STEP = 0.00 , TOTAL = 0.2 SECONDS (0.0 MIN)
> CPU UTILIZATION: STEP = 0.00%, TOTAL = 119.92%
> 197298 WORDS OF DYNAMIC MEMORY USED
> WARNING! YOU ARE USING OUTDATED VERSION OF THE FIREFLY!
> PLEASE CHECK FIREFLY HOMEPAGE FOR INFORMATION ON UPDATES!
> EXECUTION OF FIREFLY TERMINATED NORMALLY 16:05:31 LT 6-JUN-2011

>Calculations on two cores (Intel MPI 3.2.2 library):
> #!/bin/bash
> #PBS -N BAopt.inp
> #PBS -l nodes=1:ppn=2
> #PBS -l walltime=00:03:00
> #PBS -j oe
> #PBS -V
> mpi_dir=/opt/intel/mpi-rt/3.2.2/bin/
> cd $PBS_O_WORKDIR
> PATH=$PBS_O_PATH
> export I_MPI_DEVICE=rdma
> export I_MPI_DEBUG=0
> export I_MPI_FALLBACK_DEVICE=disable
> export I_MPI_PIN_MODE=lib
> export I_MPI_PIN_PROCS=0,2,1,5,3,4,6,7
> mpirun -r ssh -machinefile $PBS_NODEFILE -np `cat $PBS_NODEFILE|wc -l` ./firefly -r -f -p -stdext -ex ~/ex/ -i ~/BAopt.inp -o ~/BAopt.out -t ~/tmp/

>Folder /tmp/ and BAopt.dat and BAopt.out files cannot be created, in the initial file the following is written:
> n122:2081: dapl_cma_active: PATH_RECORD_ERR, retries(15) exhausted, DST 172.40.103.10,2079
> [0:n122] unexpected DAPL event 4008 from 1:n122
> rank 0 in job 1 n122_43384 caused collective abort of all ranks
> exit status of rank 0: killed by signal 9

>Thank you in advance!