<div dir="ltr">It is often hard to know exactly what issues are with mpi. Most often it is due to incorrect combinations of scalapack/blacs in the linking options.<div><br></div><div>The first think to check is your linking options with <a href="https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/">https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/</a>. What you have does not look exactly right to me, but I have not used your release.<div><br></div><div>If that does not work, look in case.dayfile, the log file.</div><div><br></div><div>If there is still nothing it is sometimes useful to comment out the line</div><div><div><br></div><div> CALL W2kinit</div></div><div><br></div><div>in lapw0.F, recompile then just do "x lapw0 -p". You sometimes will get more information although it is not as safe as mpi tasks can hang forever without it in some cases.</div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Oct 22, 2014 at 6:29 AM, Rémi Arras <span dir="ltr"><<a href="mailto:remi.arras@cemes.fr" target="_blank">remi.arras@cemes.fr</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<font face="Times New Roman, Times, serif"><span lang="EN-US">Dear Pr. Blaha, Dear Wien2k users,<u></u><u></u></span></font><br>
<br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">We tried to install the last version
of Wien2k (14.1) on a supercomputer and we are facing some troubles with the MPI parallel version.<u></u><u></u></span><span lang="EN-US"><u></u> <u></u></span></font>
<br>
<font face="Times New Roman, Times, serif"><span lang="EN-US"><br>
1)<span> </span>lapw0 is running correctly in sequential, but crashes systematically when the parallel option is activated (independently of the number of cores we use):<u></u><u></u></span></font><br>
<br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">><span>
</span>lapw0 -p<span> </span>(16:08:13) starting parallel lapw0 at lun. sept. 29
<u></u><u></u></span></font><font face="Times New Roman, Times, serif"><span lang="EN-US">16:08:13 CEST
2014<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">-------- .machine0 : 4 processors<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US"><span> </span>Child
id<span> </span>1 SIGSEGV<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US"><span> </span>Child
id<span> </span>2 SIGSEGV<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US"><span> </span>Child
id<span> </span>3 SIGSEGV<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US"><span> </span>Child
id<span> </span>0 SIGSEGV<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">**<span>
</span>lapw0 crashed!<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">0.029u 0.036s 0:50.91 0.0%<span>
</span>0+0k 5248+104io 17pf+0w<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">error: command<span>
</span>/eos3/p1229/remir/INSTALLATION_WIEN/14.1/lapw0para -up <u></u><u></u></span></font><font face="Times New Roman,
Times, serif"><span lang="EN-US">-c
lapw0.def<span> </span>failed<u></u><u></u></span><span lang="EN-US"><u></u> <u></u></span></font>
<br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">><span>
</span>stop error<u></u><u></u></span></font><br>
<br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">w2k_dispatch_signal(): received: Segmentation
fault<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">w2k_dispatch_signal(): received: Segmentation
fault<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US"><span> </span>Child
with myid of<span> </span>1<span>
</span>has an error<u></u><u></u></span></font><font face="Times New Roman, Times, serif"><span lang="EN-US"><span>
<br>
</span>'Unknown' - SIGSEGV<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US"><span> </span>Child
id<span> </span>1 SIGSEGV<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">application called MPI_Abort(MPI_COMM_WORLD,
0) - process 1<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">**<span>
</span>lapw0 crashed!<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">cat: No match.</span></font><font face="Times New
Roman, Times, serif"><span>0.027u
0.034s 1:33.13 0.0%<span> </span>0+0k 5200+96io 16pf+0w<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">error: command<span>
</span>/eos3/p1229/remir/INSTALLATION_WIEN/14.1/lapw0para -up -c lapw0.def<span>
</span>failed<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US"><u></u> <u></u></span>
</font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US"><br>
2) lapw2 also crashes sometimes when MPI parallelization is used. Sequential or k-parallel runs are ok, and contrary to lapw0, the error does not occur for all cases (we did not notice any problem when testing
<u></u><u></u></span></font><font face="Times New Roman, Times, serif"><span lang="EN-US">the mpi benchmark
with lapw1):</span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US"><br>
w2k_dispatch_signal(): received: Segmentation fault </span></font><font face="Times New Roman, Times, serif"><span lang="EN-US">application
called MPI_Abort(MPI_COMM_WORLD, 768) - process 0<u></u><u></u></span><span lang="EN-US"><u></u> <u></u></span></font>
<br>
<font face="Times New Roman, Times, serif"><span lang="EN-US"><br>
</span></font><font face="Times New Roman, Times, serif"><span lang="EN-US"><span lang="EN-US">Our
system is a Bullx DLC Cluster (LInux Red Hat+ Intel Ivybridge) and we use the compiler(+mkl) intel/<a href="http://14.0.2.144" target="_blank">14.0.2.144</a> and intelmpi/<a href="http://4.1.3.049" target="_blank">4.1.3.049</a></span>.<br>
The batch Scheduler is SLURM.<u></u><u></u></span></font><br>
<br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">Here are the settings and the options
we used for the installation :<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US"><u></u> <u></u></span>
</font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">OPTIONS:<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">current:FOPT:-FR -mp1 -w -prec_div
-pc80 -pad -ip -DINTEL_VML -traceback<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">current:FPOPT:-FR -mp1 -w -prec_div
-pc80 -pad -ip -DINTEL_VML -Dmkl_scalapack -traceback -xAVX<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">current:FFTW_OPT:-DFFTW3 -I/users/p1229/remir/INSTALLATION_WIEN/fftw-3.3.4-Intel_MPI/include<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">current:FFTW_LIBS:-lfftw3_mpi -lfftw3
-L/users/p1229/remir/INSTALLATION_WIEN/fftw-3.3.4-Intel_MPI/lib<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">current:LDFLAGS:$(FOPT) -L$(MKLROOT)/lib/$(MKL_TARGET_ARCH)
-pthread<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">current:DPARALLEL:'-DParallel'<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">current:R_LIBS:-lmkl_lapack95_lp64
-lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -openmp -lpthread<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">current:RP_LIBS:-mkl=cluster -lfftw3_mpi
-lfftw3 -L/users/p1229/remir/INSTALLATION_WIEN/fftw-3.3.4-Intel_MPI/lib<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">current:MPIRUN:mpirun -np _NP_ _EXEC_<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">current:MKL_TARGET_ARCH:intel64<u></u><u></u></span></font><font face="Times New Roman, Times, serif"><span lang="EN-US"><u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US"><br>
PARALLEL_OPTIONS:<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">setenv TASKSET "no"<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">setenv USE_REMOTE 1
<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">setenv MPI_REMOTE 1
<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">setenv WIEN_GRANULARITY 1<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">setenv WIEN_MPIRUN "mpirun -np _NP_
_EXEC_"<u></u><u></u></span></font><br>
<br>
<font face="Times New Roman, Times, serif"><span lang="EN-US">Any suggestions which could help us
to solve this problem would be greatly appreciated.<u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US"><u></u><u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US"><u></u>Best regards,<u></u></span></font><br>
<font face="Times New Roman, Times, serif"><span lang="EN-US"><u></u>Rémi Arras<u></u></span></font>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Professor Laurence Marks<br>Department of Materials Science and Engineering<br>Northwestern University<br><a href="http://www.numis.northwestern.edu" target="_blank">www.numis.northwestern.edu</a><div>Corrosion in 4D: <a href="http://MURI4D.numis.northwestern.edu" target="_blank">MURI4D.numis.northwestern.edu</a><br>Co-Editor, Acta Cryst A<br>"Research is to see what everybody else has seen, and to think what nobody else has thought"<br>Albert Szent-Gyorgi</div></div>
</div>