<div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif;color:#000000">A guess: your srun is setup to use openmpi or something else, not intel impi which is what you compiled for. Check what you have loaded, e.g. use "which mpirun".</div><div class="gmail_default" style="font-family:verdana,sans-serif;color:#000000"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:#000000">N.B. testing using lapw0 is simpler.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Nov 26, 2019 at 12:07 PM Hanning Chen <<a href="mailto:hchen@american.edu">hchen@american.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_4074122088704778249WordSection1">
<p class="MsoNormal"><span style="font-size:11pt">Dear WIEN2K community, <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> I am a new user of WIEN2K, and just compiled it using the following options:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u><u></u></span></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">current:FOPT:-O -FR -mp1 -w -prec_div -pc80 -pad -ip -DINTEL_VML -traceback -assume buffered_io -I$(MKLROOT)/include</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">current:FPOPT:-O -FR -mp1 -w -prec_div -pc80 -pad -ip -DINTEL_VML -traceback -assume buffered_io -I$(MKLROOT)/include</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">current:OMP_SWITCH:-qopenmp</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">current:LDFLAGS:$(FOPT) -L$(MKLROOT)/lib/$(MKL_TARGET_ARCH) -lpthread -lm -ldl -liomp5</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">current:DPARALLEL:'-DParallel'</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">current:R_LIBS:-lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">current:FFTWROOT:/home/ec2-user/FFTW338/</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">current:FFTW_VERSION:FFTW3</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">current:FFTW_LIB:lib</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">current:FFTW_LIBNAME:fftw3</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">current:LIBXCROOT:</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">current:LIBXC_FORTRAN:</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">current:LIBXC_LIBNAME:</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">current:LIBXC_LIBDNAME:</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">current:SCALAPACKROOT:$(MKLROOT)/lib/</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">current:SCALAPACK_LIBNAME:mkl_scalapack_lp64</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">current:BLACSROOT:$(MKLROOT)/lib/</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">current:BLACS_LIBNAME:mkl_blacs_intelmpi_lp64</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">current:ELPAROOT:</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">current:ELPA_VERSION:</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">current:ELPA_LIB:</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">current:ELPA_LIBNAME:</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">current:MPIRUN:srun -K -N_nodes_ -n_NP_ -r_offset_ _PINNING_ _EXEC_</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">current:CORES_PER_NODE:1</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">current:MKL_TARGET_ARCH:intel64</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">setenv TASKSET "no"</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">if ( ! $?USE_REMOTE ) setenv USE_REMOTE 1</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">if ( ! $?MPI_REMOTE ) setenv MPI_REMOTE 0</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">setenv WIEN_GRANULARITY 1</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">setenv DELAY 0.1</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">setenv SLEEPY 1</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">setenv WIEN_MPIRUN "srun -K -N_nodes_ -n_NP_ -r_offset_ _PINNING_ _EXEC_"</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1">if ( ! $?CORES_PER_NODE) setenv CORES_PER_NODE</span><span class="gmail-m_4074122088704778249apple-converted-space">
</span><span class="gmail-m_4074122088704778249s1">1</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1"># if ( ! $?PINNING_COMMAND) setenv PINNING_COMMAND "--cpu_bind=map_cpu:"</span><u></u><u></u></p>
<p class="gmail-m_4074122088704778249p1"><span class="gmail-m_4074122088704778249s1"># if ( ! $?PINNING_LIST ) setenv PINNING_LIST "0,8,1,9,2,10,3,11,4,12,5,13,6,14,7,15"</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> Then, I ran a k-point parallelization with the .machines file below, and it worked perfectly:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> granularity:1<u></u><u></u></span></p>
<p class="MsoNormal" style="text-indent:9.75pt"><span style="font-size:11pt">1:machine1<u></u><u></u></span></p>
<p class="MsoNormal" style="text-indent:9.75pt"><span style="font-size:11pt">2:machine2<u></u><u></u></span></p>
<p class="MsoNormal" style="text-indent:9.75pt"><span style="font-size:11pt">extrafine:1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> But, when I tried to parallelize it over MPI with the new .machines file:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> granularity:1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> 1:machine1 machine2<u></u><u></u></span></p>
<p class="MsoNormal" style="text-indent:9.75pt"><span style="font-size:11pt">extrafine:1<u></u><u></u></span></p>
<p class="MsoNormal" style="text-indent:9.75pt"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal" style="text-indent:9.75pt"><span style="font-size:11pt">lapw1 crashed with the error message as<u></u><u></u></span></p>
<p class="MsoNormal" style="text-indent:9.75pt"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal" style="text-indent:9.75pt"><span style="font-size:11pt">** Error in Parallel LAPW1<u></u><u></u></span></p>
<p class="MsoNormal" style="text-indent:9.75pt"><span style="font-size:11pt">**. LAPW1 STOPPED<u></u><u></u></span></p>
<p class="MsoNormal" style="text-indent:9.75pt"><span style="font-size:11pt">** check ERROR FILES!<u></u><u></u></span></p>
<p class="MsoNormal" style="text-indent:9.75pt"><span style="font-size:11pt"> SEP INFO = -21<u></u><u></u></span></p>
<p class="MsoNormal" style="text-indent:9.75pt"><span style="font-size:11pt">‘SECLR4’. -SYEVX (Scalapack/LAPACK) failed
<u></u><u></u></span></p>
<p class="MsoNormal" style="text-indent:9.75pt"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal" style="text-indent:9.75pt"><span style="font-size:11pt">Although I understand that the 21st parameter of the SYEVX subroutine is incorrect, I am not sure how to fix the problem. I actually have linked WIEN2K with NETLIB’s SCALAPACK/LAPACK/BLAS
instead of MKL. But the same error appeared again. <u></u><u></u></span></p>
<p class="MsoNormal" style="text-indent:9.75pt"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal" style="text-indent:9.75pt"><span style="font-size:11pt">Please help me out. Thanks.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">Hanning Chen, Ph.D.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">Department of Chemistry<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">American University<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">Washington, DC 20016<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
_______________________________________________<br>
Wien mailing list<br>
<a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__zeus.theochem.tuwien.ac.at_mailman_listinfo_wien&d=DwICAg&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=U_T4PL6jwANfAy4rnxTj8IUxm818jnvqKFdqWLwmqg0&m=WhQIVizHlskm1qr_hjJ-ydUa1nNmadEDmF7JYzComVg&s=EqMM1kdY2BwNqIwqgV0Ta4vW1LxanVM7jqCZyvYg3dw&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=http-3A__zeus.theochem.tuwien.ac.at_mailman_listinfo_wien&d=DwICAg&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=U_T4PL6jwANfAy4rnxTj8IUxm818jnvqKFdqWLwmqg0&m=WhQIVizHlskm1qr_hjJ-ydUa1nNmadEDmF7JYzComVg&s=EqMM1kdY2BwNqIwqgV0Ta4vW1LxanVM7jqCZyvYg3dw&e=</a> <br>
SEARCH the MAILING-LIST at: <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mail-2Darchive.com_wien-40zeus.theochem.tuwien.ac.at_index.html&d=DwICAg&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=U_T4PL6jwANfAy4rnxTj8IUxm818jnvqKFdqWLwmqg0&m=WhQIVizHlskm1qr_hjJ-ydUa1nNmadEDmF7JYzComVg&s=4y6WQbCSpPScNbNdXbfuY796kun3cl9f2BmVm8NJ3xI&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mail-2Darchive.com_wien-40zeus.theochem.tuwien.ac.at_index.html&d=DwICAg&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=U_T4PL6jwANfAy4rnxTj8IUxm818jnvqKFdqWLwmqg0&m=WhQIVizHlskm1qr_hjJ-ydUa1nNmadEDmF7JYzComVg&s=4y6WQbCSpPScNbNdXbfuY796kun3cl9f2BmVm8NJ3xI&e=</a> <br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr">Professor Laurence Marks<br>Department of Materials Science and Engineering<br>Northwestern University<br><a href="http://www.numis.northwestern.edu/" target="_blank">www.numis.northwestern.edu</a><div>Corrosion in 4D: <a href="http://www.numis.northwestern.edu/MURI" target="_blank">www.numis.northwestern.edu/MURI</a><br>Co-Editor, Acta Cryst A<br>"Research is to see what everybody else has seen, and to think what nobody else has thought"<br>Albert Szent-Gyorgi</div></div></div>