<html><body><div style="color:#000; background-color:#fff; font-family:times new roman, new york, times, serif;font-size:12pt"><div><span><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">Dear All,</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; min-height: 13px; font-family: Helvetica; font-size: medium; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: 16px; ">I compiled the latest version of the code on a cluster made up of several Dual Core AMD Opteron nodes by ifort and mpif90.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal
Calibri; font-family: Helvetica; font-size: 16px; ">There is no any error or warning in the SRC_*/compile.msg files. The code runs well on the nodes, if we only parallelize the k-points by for example the following .machines file:</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; "> lapw0:node3 node20</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">1:node3</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">1:node20</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri;
font-family: Helvetica; font-size: medium; ">granularity:1</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">extrafine:1</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; min-height: 13px; font-family: Helvetica; font-size: medium; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">But, the program stops with the following error: </div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">ifort: command line warning #10159: invalid argument for option '-m'</div><div
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">ifort: command line error: option '-n' is ambiguous</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">once we use the fine grained parallelization by for example the following .machines file:</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; "> lapw0:node3:2 node20:2</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">1:node3:2</div><div style="margin-top: 0px; margin-right: 0px;
margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">1:node20:2</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">granularity:1</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">extrafine:1</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; min-height: 13px; font-family: Helvetica; font-size: medium; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">I have used l_cprof_p_11.1.073_intel64,
fftw-2.1.5, mpich2 for compiling the code, and the following settings.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">-----------------------------------------</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">Current settings:</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; "> O Compiler options: -FR -mp1 -w -prec_div -pc80 -pad -ip -DINTEL_VML -traceback</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri;
font-family: Helvetica; font-size: medium; "> L Linker Flags: $(FOPT) -L/home/softs/intel/ifort11/mkl/lib/em64t -pthread</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; "> P Preprocessor flags '-DParallel'</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; "> R R_LIB (LAPACK+BLAS): -lmkl_lapack -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lpthread -lguide</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size:
medium; ">-------------------------------------------</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; "> RP RP_LIB(SCALAPACK+PBLAS): -lmkl_scalapack_lp64 -lmkl_solver_lp64 -lmkl_blacs_lp64 -L/home/softs/mpich2/lib -lmpich -L/home/softs/fftw-2.1.5/mpi/.libs/ -lfftw_mpi -L/home/softs/fftw-2.1.5/fftw/.libs/ -lfftw $(R_LIBS)</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; "> FP FPOPT(par.comp.options): -FR -mp1 -w -prec_div -pc80 -pad -ip -DINTEL_VML -traceback -I/home/softs/mpch2/include</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri;
font-family: Helvetica; font-size: medium; "> MP MPIRUN commando : /home/softs/mpich2/bin/mpif90 -machinefile _HOSTS_ -n _NP_ _EXEC_</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">---------------------------------------------</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">The parallel_options is: </div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">---------------------------------------------</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;
font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">setenv USE_REMOTE 1</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">setenv MPI_REMOTE 1</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">setenv WIEN_GRANULARITY 1</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">setenv WIEN_MPIRUN "/home/softs/mpich2/bin/mpif90 -machinefile _HOSTS_ -n _NP_ _EXEC_"</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 10px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica;
font-size: medium; ">-------------------------------</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 10px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">I changed the mpif90 to mpirun only in the parallel options (just as a test) but I did not recompile the code by mpirun.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 10px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">The result is as follows:</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 10px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">LAPW0 END</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 10px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; "> LAPW0
END</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 10px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">Fatal error in PMPI_Comm_size: Invalid communicator, error stack:</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 10px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">PMPI_Comm_size(111): MPI_Comm_size(comm=0x5b, size=0x8aa96c) failed</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 10px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">PMPI_Comm_size(69).: Invalid communicator</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 10px; margin-left: 0px; font: normal normal normal 11px/normal Arial; min-height: 12px; font-family: Helvetica; font-size: medium; "><br></div><div
style="margin-top: 0px; margin-right: 0px; margin-bottom: 10px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">real<span class="Apple-tab-span" style="white-space: pre; "> </span>0m0.050s</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 10px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">user<span class="Apple-tab-span" style="white-space: pre; "> </span>0m0.010s</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 10px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">sys<span class="Apple-tab-span" style="white-space: pre; "> </span>0m0.038s</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 10px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium;
">test.scf1_1: No such file or directory.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 10px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">FERMI - Error</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 10px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">cp: cannot stat `.in.tmp': No such file or directory</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 10px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">rm: cannot remove `.in.tmp': No such file or directory</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 10px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">rm: cannot remove `.in.tmp1': No such file or
directory</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 10px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; min-height: 13px; font-family: Helvetica; font-size: medium; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 10px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">Similar error was occurred when I used mpiexec in the parallel options without recompiling the code. </div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 10px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">I found that the "Invalid communicator" originates from incompatible mpi.h of mpirun or mpiexec with that of mpif90.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 10px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family:
Helvetica; font-size: medium; ">So I changed back it to mpif90.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 10px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">Since I guessed that the problem originates from the version of mpi, I tried different versions of mpi, i.e., mpich2-1.0.6, mpich2-1.3.1, mpich2-1.4, openmpi-1.4.2.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 10px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; font-family: Helvetica; font-size: medium; ">Any comment on why the code is compiled with no error or warning, but it stops with error is highly appreciated. </div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 10px; margin-left: 0px; font: normal normal normal 11px/normal Calibri; min-height: 13px; font-family: Helvetica; font-size: 16px; ">Is there any restrictions for compiling the code
by mpif90 on AMD systems, as discussed above?</div></span></div><div> </div><div>Sincerely yours,<br>S. Jalali<br>/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/<br>Saeid Jalali Asadabadi,<br>Department of Physics, Faculty of Science,<br>University of Isfahan (UI), Hezar Gerib Avenue,<br>81744 Isfahan, Iran.<br>Phones:<br>Dep. of Phys. :+98-0311-793 2435<br>Office :+98-0311-793 4776<br>Fax No. :+98-0311-793 4800<br>E-mail :<a rel="nofollow" target="_blank" href="mailto:sjalali@phys.ui.ac.ir">sjalali@phys.ui.ac.ir</a><br> :<a rel="nofollow" target="_blank" href="mailto:sjalali@sci.ui.ac.ir">sjalali@sci.ui.ac.ir</a><br>
:<a rel="nofollow" target="_blank" href="mailto:sjalali@mailaps.org">sjalali@mailaps.org</a><br> :<a rel="nofollow" target="_blank" href="mailto:saeid.jalali.asadabadi@gmail.com">saeid.jalali.asadabadi@gmail.com</a><br> :<a rel="nofollow" target="_blank" href="mailto:s_jalali_a@yahoo.com">s_jalali_a@yahoo.com</a><br>Homepage :<a rel="nofollow" target="_blank" href="http://sci.ui.ac.ir/%7Esjalali">http://sci.ui.ac.ir/~sjalali</a><br>www :<a rel="nofollow" target="_blank" href="http://www.ui.ac.ir/">http://www.ui.ac.ir</a><br>/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/<br><br></div> </div></body></html>