[Wien] MPI stuck at lapw0

[Luigi Maduro - TNW]

Dear WIEN2k users,

I have the following problem. I am trying to do parallel computing on a cluster. Whenever I run a job on the cluster on one node both the  MPI and k-point parallelization work fine. However, when I try to go to several nodes the job does not do anything. The script just gets stuck on lapw0 whenever MPI is implemented, the k-point parallelization does not give a problem when running on multiple nodes. Additionally, if instead I try to run a job where lapw0 is running parallel only on one node on multiple processors, but lapw1 and lapw2 are run on multiple nodes, then again there is no problem.
I do not get an error while running lapw0 parallel over multiple nodes: the job doesn't do anything. The assigned nodes are scheduled to the job, but the load on the nodes stays at 0%. When I forcibly stop the job then I get these following errors:



[mpiexec at n05-38] HYDU_sock_write (../../utils/sock/sock.c:417): write error (Bad file descriptor)

[mpiexec at n05-38] HYD_pmcd_pmiserv_send_signal (../../pm/pmiserv/pmiserv_cb.c:252): unable to write data to proxy

[mpiexec at n05-38] ui_cmd_cb (../../pm/pmiserv/pmiserv_pmci.c:174): unable to send signal downstream

[mpiexec at n05-38] HYDT_dmxu_poll_wait_for_event (../../tools/demux/demux_poll.c:76): callback returned error status

[mpiexec at n05-38] HYD_pmci_wait_for_completion (../../pm/pmiserv/pmiserv_pmci.c:500): error waiting for event

[mpiexec at n05-38] main (../../ui/mpich/mpiexec.c:1130): process manager error waiting for completion

In the example above I tried to run a job on nodes n05-32 and n05-38.
The operating system on the cluster is CentOS 7<http://www.centos.org/>. The cluster consists of a master node where Mau and Torque are running (PBS implementation). The cluster is set up so that rsh can be used instead of ssh.
I am using intel's parallel studio 2016:

My .machines file looked like this:

n05-32:10
n05-38:10
lapw0:n05-32:10 n05-38:10
extrafine=1
granularity=1

In the script I submit for the jobs I source my .bashrc file:
source /home/.bashrc


The .bashrc file has the following lines:
source /opt/ud/intel_xe_2016/parallel_studio_xe_2016.3.067/compilers_and_libraries_2016/linux/mkl/bin/mklvars.sh intel64

source /opt/ud/intel_xe_2016/parallel_studio_xe_2016.3.067/compilers_and_libraries_2016/linux/bin/compilervars.sh intel64

source /opt/ud/intel_xe_2016/parallel_studio_xe_2016.3.067/compilers_and_libraries_2016/linux/bin/iccvars.sh intel64

source /opt/ud/intel_xe_2016/parallel_studio_xe_2016.3.067/compilers_and_libraries_2016/linux/mpi/intel64/bin/mpivars.sh intel64



export PATH=$PATH:/opt/ud/intel_xe_2016/parallel_studio_xe_2016.3.067/compilers_and_libraries_2016/linux/mpi/intel64/bin

export PATH=$PATH:/opt/ud/intel_xe_2016/parallel_studio_xe_2016.3.067/compilers_and_libraries_2016/linux/mkl/include

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/ud/intel_xe_2016/parallel_studio_xe_2016.3.067/compilers_and_libraries_2016/linux/mpi/intel64/lib

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/ud/intel_xe_2016/parallel_studio_xe_2016.3.067/compilers_and_libraries_2016/linux/mkl/lib/intel64




Along with the rest of the alias' and environment variables that ./userconfig_lapw sets up.
Some more parameters:
System: LI
ifort compiler
icc c compiler

mpiifort compiler

Compiler options: -O1 -FR -mp1 -w -prec_div -pc80 -pad -ip -DINTEL_VML -traceback -assume buffered_io -I/opt/ud/intel_xe_2016/parallel_studio_xe_2016.3.067/compilers_and_libraries_2016/linux/mkl/include



Linker flags: $(FOPT) -L/opt/ud/intel_xe_2016/parallel_studio_xe_2016.3.067/compilers_and_libraries_2016/linux/mkl/lib/intel64 -lpthread



Preprocessors flags: '-DParallel'



R_LIBS: -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -openmp -lpthread -liomp5
SCALAPACK: -L/opt/ud/intel_xe_2016/parallel_studio_xe_2016.3.067/compilers_and_libraries_2016/linux/mkl/lib/intel64 -lmkl_scalapack_lp64 -L/opt/ud/intel_xe_2016/parallel_studio_xe_2016.3.067/compilers_and_libraries_2016/linux/mkl/lib/intel64 -lmkl_blacs_intelmpi_lp64


And my parallel options file looks like this:

setenv TASKSET "no"
if ( ! $?USE_REMOTE ) setenv USE_REMOTE 1
if ( ! $?MPI_REMOTE ) setenv MPI_REMOTE 1
setenv WIEN_GRANULARITY 1
setenv DELAY 0.1
setenv SLEEPY 1
setenv WIEN_MPIRUN "/opt/ud/intel_xe_2016/parallel_studio_xe_2016.3.067/compilers_and_libraries_2016/linux/mpi/intel64/bin/mpirun -ppn $PBS_NUM_PPN -np _NP_ -machinefile _HOSTS_ _EXEC_"

setenv CORES_PER_NODE 20



I downloaded my fftw package and configured it with the options F77=ifort CC=icc MPICC=mpiicc

Similarly for the LIBXC package: FC=ifort CC=icc



Any help is appreciated.
Luigi Maduro




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://zeus.theochem.tuwien.ac.at/pipermail/wien/attachments/20171017/73aa72ef/attachment.html>