[Wien] some comments on parallel execution of wien2k

Wed Dec 23 13:08:12 CET 2009

Dear wien2k users and developers,

I would like to post few comments on running parallel version of wien2k 
on a distributed memory cluster. I'm using the most recent version of 
wien2k (09.2) on a Linux-based cluster with 805 HP ProLiant DL140 G3 
nodes, each node consisting of Intel Xeon E5345 Quad Core Processor 2.33 
GHz, 4 MB Level 2 cache, interconnected by Next generation Infiniband 
interconnect. Operating system is CentOS 5 64-bit Linux and resource 
manager is SLURM. I compiled source code with Intel compilers (ifort 
10.1.017), Intel built OpenMPI (mpif90 1.2.7) and linked with MKL 
(10.0.1.014), FFTW (2.1.5) and corresponding OpenMPI libs.

My first comment concerns the implementation of the MPI on fine grain 
parallelization. Within the current version of wien2k, the module 
lap2w_mpi crashes if N_noneq_atoms (number of nonequivalent atoms in 
case.struc file) is not a multiple of N_cpus (number of processors to 
run lapw2_mpi). This strange behavior was reported in a recent post by 
Duy Le with the subject “[Wien] MPI problem for LAPW2” 
(http://zeus.theochem.tuwien.ac.at/pipermail/wien/2009-September/012042.html). 
He noticed that for a system consisting of 21 (nonequivalent) atoms the 
program runs only for 3 or 7 cpus. He managed to cure the problem by 
setting lapw2_vector_split:$N_cpus, but without a reasonable 
explanation. However, one can get a hint by looking at lap2w source 
files and the output of lapw2_mpi. Let's consider, for example, the 
cd16te15sb.struct from $WIENROOT/example_struct_files, which describes a 
structure with 5 nonequivalent atoms. Let's run it on a computer node 
with 8 cpus with the following .machines file:
granularity:1
1:n246:8
lapw0:n246:8
extrafine:1

Here are some lines from the resulting case.dayfile:
 > lapw0 -p (19:41:09) starting parallel lapw0 at Tue Dec 22 19:41:09 
CET 2009
-------- .machine0 : 8 processors
mpirun --verbose -np 8 --hostfile .machine0 $WIENROOT/lapw0_mpi lapw0.def
Tue Dec 22 19:41:27 CET 2009 -> all processes done.
….................................................................................. 

 > lapw1 -c -p (19:41:28) starting parallel lapw1 at Tue Dec 22 19:41:28 
CET 2009
-> starting parallel LAPW1 jobs at Tue Dec 22 19:41:28 CET 2009
1 number_of_parallel_jobs
-------- .machine1 : 8 processors : weight 1
mpirun --verbose -np 8 --hostfile .machine1 $WIENROOT/lapw1c_mpi 
lapw1_1.def
waiting for all processes to complete
Tue Dec 22 19:48:26 CET 2009 -> all processes done.
…..................................................................................... 

 > lapw2 -c -p (19:48:28) running LAPW2 in parallel mode
running parallel lapw2
mpirun --verbose -np 8 --hostfile .machine1 $WIENROOT/lapw2c_mpi 
lapw2_1.def 1
sleeping for 1 seconds
waiting for processes:
** LAPW2 crashed!
…..................................................................................... 

The job crashed with the following error message:
[n246:15992] *** An error occurred in MPI_Comm_split
[n246:15992] *** on communicator MPI_COMM_WORLD
[n246:15992] *** MPI_ERR_ARG: invalid argument of some other kind
[n246:15992] *** MPI_ERRORS_ARE_FATAL (goodbye)

Now, if one take a look at case.output2_1_proc_n (n=1,2,..,7), one see 
the following header (here is the case of case.output2_1_proc_1):

init_parallel_2 1 8 1 8 2
MPI run on 8 processors in MPI_COMM_WORLD
8 processors in MPI_vec_COMM (atoms splitting)
1 processors in MPI_atoms_COMM (vector splitting)

myid= 1
myid_atm= 1
myid_vec= 1

time in recpr: 0.820000000000000

One can find the following lines in the lapw2.F source file (lines 
129-137):
#ifdef Parallel
write(6,*) 'MPI run on ',npe,' processors in MPI_COMM_WORLD'
write(6,*) ' ',npe_atm,' processors in MPI_vec_COMM (atoms splitting)'
write(6,*) ' ',npe_vec,' processors in MPI_atoms_COMM (vector splitting)'
write(6,*) ' myid= ',myid
write(6,*) ' myid_atm= ',myid
write(6,*) ' myid_vec= ',myid
write(6,*) ' '
#endif
which generate this output.

If I correctly understand, npe is the total number of cpus, npe_atm is 
the number of cpus for parallelization over atoms, and npe_vec the 
number of cpus for additional parallelization of the density over 
vectors (I think that MPI_vec_COMM and MPI_atoms_COMM should be swapped).

One can also find the following lines (306-311) in the l2main.F file:
! ---------------------------------
! START LOOP FOR ALL ATOMS
! ---------------------------------

non_equiv_loop: do jatom_pe=1,nat,npe_atm
jatom=jatom_pe+myid_atm

from which I can understand that the loop is over nonequivalent atoms 
nat with step npe_atom.

Now let's do some changes in the lapw2para to run lapw2_mpi on 5 cpus, 
and take a look at case.dayfile and case.output2_1_proc_1 files.
Here are lines from case.dayfile:
 > lapw0 -p (20:08:14) starting parallel lapw0 at Tue Dec 22 20:08:14 
CET 2009
Tue Dec 22 20:08:14 CET 2009 -> Setting up case Cd16Te15Sb for parallel 
execution
-------- .machine0 : 8 processors
mpirun --verbose -np 8 --hostfile .machine0 $WIENROOT/lapw0_mpi lapw0.def
Tue Dec 22 20:08:33 CET 2009 -> all processes done.
…............................................................................. 

 > lapw1 -c -p (20:08:34) starting parallel lapw1 at Tue Dec 22 20:08:34 
CET 2009
-> starting parallel LAPW1 jobs at Tue Dec 22 20:08:34 CET 2009
mpirun --verbose -np 8 --hostfile .machine1 $WIENROOT/lapw1c_mpi 
lapw1_1.def
waiting for all processes to complete
Tue Dec 22 20:15:32 CET 2009 -> all processes done.
…................................................................................................... 

 > lapw2 -c -p (20:15:33) running LAPW2 in parallel mode
machines: n383
running parallel lapw2
mpirun --verbose -np 5 --hostfile .machine1 $WIENROOT/lapw2c_mpi 
lapw2_1.def 1
sleeping for 1 seconds
waiting for processes:
n383 0.014u 0.008s 0:51.32 0.0% 0+0k 0+0io 0pf+0w
…................................................
:ENERGY convergence: 0 0 .0004326450000000
:CHARGE convergence: 1 0.001 -.0000259
ec cc and fc_conv 1 1 1
 > stop

This time job terminated successfully, and first lines of 
case.output2_1_proc_1 read as:
init_parallel_2 1 5 1 5 2
MPI run on 5 processors in MPI_COMM_WORLD
5 processors in MPI_vec_COMM (atoms splitting)
1 processors in MPI_atoms_COMM (vector splitting)

myid= 1
myid_atm= 1
myid_vec= 1

time in recpr: 0.820000000000000

:POS002: AT.NR. -2 POSITION = 0.12426 0.12426 0.12426 MULTIPLICITY = 4
….................................................................................................................................... 

Now, let's see what is going on when lapw2_vector_split:N_cpus is set in 
.machines file:
granularity:1
1:n21:8
lapw2_vector_split:8
lapw0:n21:8
extrafine:1

One can read the following lines from case.dayfile:
 > lapw2 -c -p (21:06:51) running LAPW2 in parallel mode
machines: n21
running parallel lapw2
mpirun --verbose -np 8 --hostfile .machine1 
/home/x_serar/wien2k/09.2/openmpi/lapw2c_mpi lapw2_1.def 1
sleeping for 1 seconds
waiting for processes:
n21 0.027u 0.012s 1:26.55 0.0% 0+0k 0+0io 0pf+0w
…....................................................................................................... 

:ENERGY convergence: 0 0 .0000601700000000
:CHARGE convergence: 1 0.001 -.0006113
ec cc and fc_conv 1 1 1
 > stop

The first lines of case.output2_1_proc_1 read as:
init_parallel_2 1 8 8 1 1
MPI run on 8 processors in MPI_COMM_WORLD
1 processors in MPI_vec_COMM (atoms splitting)
8 processors in MPI_atoms_COMM (vector splitting)

myid= 1
myid_atm= 1
myid_vec= 1

time in recpr: 0.810000000000000
0 0.191606E+00 -0.475342E+00 -0.217499E+00 -0.295484E+00 0.795881E-01 4 4 4
…....................................................................................................................................................... 

That is, there is no atom splitting and npe_atm=1 is a divisor of the 
number of nonequivalent atoms. This result npe_atm=1 becomes clear if 
one take a look at modules.F, SUBROUTINE init_parallel_2 (line 78):
................................................................................... 

npe_atm=npe/npe_vec
…............................................................................... 

Thus, the crash of lapw2_mpi is not related to memory issues, but to the 
way the parallelization is implemented. My analysis is, of course, 
superficial, and I can not say whether there is a bug in the lapw2_mpi 
module. But I think that this issue requires some attention from 
developers.

My second comment is that you do not need to connect through ssh to 
allocated processors on different computer nodes in order to run 
lapw1(c) or lapw2(c) (the case of parallelization over k-points). You 
can run your parallel processes by invoking mpirun.
First, set up "setenv WIEN_MPIRUN 'mpirun -np _NP_ --hostfile _HOSTS_ 
_EXEC_'” in $WIENROOT/parallel_options.
Second, instead of line “(cd $PWD;$t $exe ${def}_$loop.def;rm -f 
.lock_$lockfile[$p]) >>.time1_$loop &” in lapw1para (line 406) use the 
following two lines:
“set ttt=(`echo $mpirun | sed -e "s^_NP_^$number_per_job[$p]^" -e 
"s^_HOSTS_^.machine$p^" -e "s^_EXEC_^$WIENROOT/${exe} ${def}_$loop.def^"`)”
and
“(cd $PWD;$t $ttt;rm -f .lock_$lockfile[$p]) >>.time1_$loop &”

similar to mpi execution.
In the same fashion, in lapw2para instead of line 314 “(cd $PWD;$t $exe 
${def}_${loop}.def $loop;rm -f .lock_$lockfile[$p]) >>.time2_$loop &” 
use the following 2 lines:
“set ttt=(`echo $mpirun | sed -e "s^_NP_^$number_per_job2[$loop]^" -e 
"s^_HOSTS_^.machine$mach[$loop]^" -e "s^_EXEC_^$WIENROOT/${exe} 
${def}_$loop.def $loop^"`)”
and
“(cd $PWD;$t $ttt $vector_split;rm -f .lock_$lockfile[$p]) 
 >>.time2_$loop &”.

I hope you will find these comments useful :) .

Regards and Marry Christmas,
Sergiu Arapan