<html><body><span style="display:block;" class="xfm_93388870"><blockquote class="xfmc1" style="border-left:1px solid rgb(204, 204, 204);margin:0px 0px 0px 0.8ex;padding-left:1ex;"><pre>Dear Prof. Blaha,<div><br/></div><div>I'm using intel mpi <span data-ukrnet-code="2019">2019</span>.3.199</div><div>the scalapack and blacs libs are located in the intel compilers_and_libraries_<span data-ukrnet-code="2019">2019</span>.3.199 folder </div><div><br/></div><div>OPTIONS file:</div><div><div style="font-family:Arial, sans-serif;white-space:normal;">current:FOPT:-O1 -FR -mp1 -w -prec_div -pc80 -pad -ip -DINTEL_VML -traceback -assume buffered_io -I$(MKLROOT)/include</div><div style="font-family:Arial, sans-serif;white-space:normal;">current:FPOPT:-O1 -FR -mp1 -w -prec_div -pc80 -pad -ip -DINTEL_VML -traceback -assume buffered_io -I$(MKLROOT)/include</div><div style="font-family:Arial, sans-serif;white-space:normal;">current:LDFLAGS:$(FOPT) -L$(MKLROOT)/lib/intel64 -lpthread -lm -ldl -liomp5</div><div style="font-family:Arial, sans-serif;white-space:normal;">current:DPARALLEL:-DParallel</div><div style="font-family:Arial, sans-serif;white-space:normal;">current:R_LIBS:-lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core</div><div style="font-family:Arial, sans-serif;white-space:normal;">current:FFTWROOT:/gpfs/home/ser/Install/FFTW/</div><div style="font-family:Arial, sans-serif;white-space:normal;">current:FFTW_VERSION:FFTW3</div><div style="font-family:Arial, sans-serif;white-space:normal;">current:FFTW_LIB:lib</div><div style="font-family:Arial, sans-serif;white-space:normal;">current:FFTW_LIBNAME:fftw3</div><div style="font-family:Arial, sans-serif;white-space:normal;">current:LIBXCROOT:/gpfs/home/ser/Install/LIBXC/</div><div style="font-family:Arial, sans-serif;white-space:normal;">current:LIBXC_FORTRAN:xcf03</div><div style="font-family:Arial, sans-serif;white-space:normal;">current:LIBXC_LIBNAME:xc</div><div style="font-family:Arial, sans-serif;white-space:normal;">current:LIBXC_LIBDNAME:lib/</div><div style="font-family:Arial, sans-serif;white-space:normal;">current:SCALAPACKROOT:/gpfs/softs/cluster/intel/psxe/<span data-ukrnet-code="2019">2019</span>.3/compilers_and_libraries_<span data-ukrnet-code="2019">2019</span>.3.199/linux/mkl/lib/</div><div style="font-family:Arial, sans-serif;white-space:normal;">current:SCALAPACK_LIBNAME:mkl_scalapack_lp64</div><div style="font-family:Arial, sans-serif;white-space:normal;">current:BLACSROOT:/gpfs/softs/cluster/intel/psxe/<span data-ukrnet-code="2019">2019</span>.3/compilers_and_libraries_<span data-ukrnet-code="2019">2019</span>.3.199/linux/mkl/lib/</div><div style="font-family:Arial, sans-serif;white-space:normal;">current:BLACS_LIBNAME:mkl_blacs_intelmpi_lp64</div><div style="font-family:Arial, sans-serif;white-space:normal;">current:ELPAROOT:</div><div style="font-family:Arial, sans-serif;white-space:normal;">current:ELPA_VERSION:</div><div style="font-family:Arial, sans-serif;white-space:normal;">current:MPIRUN:srun -K -N_nodes_ -n_NP_ -r_offset_ _PINNING_ _EXEC_ (changed to mpirun)</div><div style="font-family:Arial, sans-serif;white-space:normal;">current:CORES_PER_NODE:1</div><div style="font-family:Arial, sans-serif;white-space:normal;">current:MKL_TARGET_ARCH:intel64</div><div style="font-family:Arial, sans-serif;white-space:normal;"><br/></div><div style="font-family:Arial, sans-serif;white-space:normal;"><br/></div><div style="font-family:Arial, sans-serif;white-space:normal;">part of .bashrc</div><div style="font-family:Arial, sans-serif;white-space:normal;"><br/></div><div style="font-family:Arial, sans-serif;white-space:normal;"><div>module load gcc/7.3.0</div><div>module add compiler/intel/<span data-ukrnet-code="2019">2019</span>.3.199</div><div>module load mpi/intel/<span data-ukrnet-code="2019">2019</span>.3.199</div><div>source /gpfs/softs/cluster/intel/psxe/<span data-ukrnet-code="2019">2019</span>.3/compilers_and_libraries_<span data-ukrnet-code="2019">2019</span>.3.199/linux/bin/compilervars.sh intel64</div></div><div style="font-family:Arial, sans-serif;white-space:normal;">source /gpfs/softs/cluster/intel/psxe/<span data-ukrnet-code="2019">2019</span>.3/compilers_and_libraries_<span data-ukrnet-code="2019">2019</span>.3.199/linux/mpi/intel64/bin/mpivars.sh<br/></div><div style="font-family:Arial, sans-serif;white-space:normal;"><br/></div><div style="font-family:Arial, sans-serif;white-space:normal;">in the interactive mode</div><div style="font-family:Arial, sans-serif;white-space:normal;"><pre>mpirun -np 4 $WIENROOT/lapw0_mpi lapw0.def</pre></div><div style="font-family:Arial, sans-serif;white-space:normal;">results in LAPW0 END</div><div style="font-family:Arial, sans-serif;white-space:normal;"><br/></div><div style="font-family:Arial, sans-serif;white-space:normal;">Actually, after I commented the following line in my script </div><div style="font-family:Arial, sans-serif;white-space:normal;">"if($natom < $nproc) set nproc0=$natom"</div><div style="font-family:Arial, sans-serif;white-space:normal;">the "permission denied" error disappeared and mpi starts with the following output:</div><div style="font-family:Arial, sans-serif;white-space:normal;"><br/></div><div style="font-family:Arial, sans-serif;white-space:normal;"><div>32 nodes for this job: n073 n073 n073 n073 n073 n073 n073 n073 n073 n073 n073 n073 n073 n073 n073 n073 n074 n074 n074 n074 n074 n074 n074 n074 n074 n074 n074 n074 n074 n074 n074 n074</div><div> LAPW0 END</div><div>[1] Done mpirun -n 32 -machinefile .machine0 /gpfs/home/ser/wienroot_v18/lapw0_mpi lapw0.def >> .time00</div><div>Force-convergence not possible. Forces not present.</div><div> LAPW1 END</div><div>[1] + Done ( cd $PWD; $t $ttt; rm -f .lock_$lockfile[$p] ) >> .time1_$loop</div><div> LAPW1 END</div><div>[1] + Done ( cd $PWD; $t $ttt; rm -f .lock_$lockfile[$p] ) >> .time1_$loop</div><div>LAPW2 - FERMI; weights written</div><div>LAPW2 - FERMI; weights written</div><div> CORE END</div><div> CORE END</div><div> MIXER END</div></div><div style="font-family:Arial, sans-serif;white-space:normal;"><br/></div><div style="font-family:Arial, sans-serif;white-space:normal;">At the same time in dayfile:</div><div style="font-family:Arial, sans-serif;white-space:normal;"><div>Intel MKL ERROR: Parameter 3 was incorrect on entry to DGEMM .</div><div><br/></div><div>Intel MKL ERROR: Parameter 3 was incorrect on entry to DGEMM .</div><div><br/></div><div>Intel MKL ERROR: Parameter 3 was incorrect on entry to DGEMM .</div><div><br/></div><div>Intel MKL ERROR: Parameter 8 was incorrect on entry to DGEMM .</div><div><br/></div><div>Intel MKL ERROR: Parameter 8 was incorrect on entry to DGEMM .</div></div><div style="font-family:Arial, sans-serif;white-space:normal;">....</div><div style="font-family:Arial, sans-serif;white-space:normal;"><br/></div><div style="font-family:Arial, sans-serif;white-space:normal;">and in scf file</div><div style="font-family:Arial, sans-serif;white-space:normal;"><div>:seclit_par: estimate of singular value, factor: 0.3125E-01 0.1000E-14</div><div>:WARN :seclit_par-stability trick active for: eigenvalue, sproj_ii 1 0.8207E-20</div><div>:WARN :seclit_par-stability trick active for: eigenvalue, sproj_ii 2 0.6228E-19</div><div>:WARN :seclit_par-stability trick active for: eigenvalue, sproj_ii 3 0.6073E-19</div><div>:WARN :seclit_par-stability trick active for: eigenvalue, sproj_ii 4 0.6256E-19</div><div>:WARN :seclit_par-stability trick active for: eigenvalue, sproj_ii 5 0.9136E-19</div><div>:WARN :seclit_par-stability trick active for: eigenvalue, sproj_ii 6 0.9098E-19</div><div>:WARN :seclit_par-stability trick active for: eigenvalue, sproj_ii 7 0.7724E-19</div><div>:WARN :seclit_par-stability trick active for: eigenvalue, sproj_ii 8 0.7724E-19</div><div>:WARN :seclit_par-stability trick active for: eigenvalue, sproj_ii 9 0.7534E-19</div><div>:WARN :seclit_par-stability trick active for: eigenvalue, sproj_ii 10 0.2265E-19</div><div>:WARN :seclit_par-stability trick active for: eigenvalue, sproj_ii 11 0.2059E-19</div><div>:WARN :seclit_par-stability trick active for: eigenvalue, sproj_ii 12 0.2059E-19</div><div>:WARN :seclit_par-stability trick active for: eigenvalue, sproj_ii 13 0.8401E-18</div><div>:WARN :seclit_par-stability trick active for: eigenvalue, sproj_ii 14 0.8294E-18</div><div>:WARN :seclit_par-stability trick active for: eigenvalue, sproj_ii 15 0.1019E-19</div><div>:WARN :seclit_par-stability trick active for: eigenvalue, sproj_ii 16 0.2041E-19</div></div><div style="font-family:Arial, sans-serif;white-space:normal;"><br/></div><div style="font-family:Arial, sans-serif;white-space:normal;"><br/></div><div style="font-family:Arial, sans-serif;white-space:normal;">This messages are absent in case of k-point parallelization</div><div style="font-family:Arial, sans-serif;white-space:normal;"><br/></div></div>
</pre></blockquote> </span></body></html>