<div dir="ltr">Dear Experts<div><br></div><div>I just installed Wien2k_16 on a sge cluster (<span style="white-space:pre-wrap;color:rgb(0,0,0)">linuxifc) </span>with 40 nodes with each node having 16 core and each core has 4GB RAM (~2GB/ processor), <span style="color:rgb(0,0,0);font-family:"times new roman";text-align:justify">40 Gbps Infiniband interconnect</span>. I used "mpiifort" and "mpiicc" compiler with scalapck, blas, fftd3 and blacs library (without ELPA and LIBXC-3.0.0). I also specified number of core (16) during configuration for each node (the compiler options are specified at the bottom or email).</div><div><br></div><div>Now I have submitted the job using the sge script:</div><div><br></div><div><a href="http://susi.theochem.tuwien.ac.at/reg_user/faq/sge.job">http://susi.theochem.tuwien.ac.at/reg_user/faq/sge.job</a></div><div><br></div><div>with <span style="color:rgb(0,0,0);white-space:pre-wrap">set mpijob=2 instead of </span><span style="color:rgb(0,0,0);white-space:pre-wrap">set mpijob=1.</span></div><div><br></div><div><br></div><div>I spacified</div><div><div><font color="#000000"> PARAMETER (NMATMAX= 19000)</font></div><div><font color="#000000"> PARAMETER (NUME= 6000)</font></div></div><div><font color="#000000"><br></font></div><div><font color="#000000">Now I have few queries:</font></div><div><font color="#000000">(1) </font>is it ok with mpiifort or mpicc or it should have mpifort or mpicc??</div><div>(2) how to know that job is running with mpi parallelization?</div><div><br></div><div><br></div><div>the basic outputs are:</div><div><br></div><div><div dir="ltr"><div dir="ltr"><font color="#000000">[bhamu@gu CuGaO2]$ testpara1_lapw</font></div><div dir="ltr"><font color="#000000">.processes: No such file or directory.</font></div><div dir="ltr"><font color="#000000">(standard_in) 1: syntax error</font></div><div dir="ltr"><font color="#000000"><br></font></div><div dir="ltr"><font color="#000000">#####################################################</font></div><div dir="ltr"><font color="#000000"># TESTPARA1 #</font></div><div dir="ltr"><font color="#000000">#####################################################</font></div><div dir="ltr"><font color="#000000"><br></font></div><div dir="ltr"><font color="#000000">Tue Jan 17 22:14:57 IST 2017</font></div><div dir="ltr"><font color="#000000"><br></font></div><div dir="ltr"><font color="#000000"> lapw1para was not yet executed</font></div><div style="color:rgb(0,0,0)"><br></div></div><div dir="ltr"><span style="color:rgb(0,0,0)">the *.err file seems as:</span><br></div><div dir="ltr"><div dir="ltr" style="color:rgb(0,0,0)">LAPW0 END</div><div dir="ltr" style="color:rgb(0,0,0)"> ORB END</div><div dir="ltr" style="color:rgb(0,0,0)"> ORB END</div><div dir="ltr" style="color:rgb(0,0,0)"> LAPW1 END</div><div dir="ltr" style="color:rgb(0,0,0)"> LAPW2 END</div><div dir="ltr" style="color:rgb(0,0,0)">cp: cannot stat `CuGaO2.scfdmup': No such file or directory >>> why this is error? I want to overcome this.</div><div dir="ltr" style="color:rgb(0,0,0)"> CORE END</div><div dir="ltr" style="color:rgb(0,0,0)"> CORE END</div><div dir="ltr" style="color:rgb(0,0,0)"> MIXER END</div><div dir="ltr" style="color:rgb(0,0,0)"><br></div><div style="color:rgb(0,0,0)">The :log file</div><div style="color:rgb(0,0,0)"><br></div><div><div><font color="#000000">Tue Jan 17 22:16:14 IST 2017> (x) lapw0</font></div><div><font color="#000000">Tue Jan 17 22:16:17 IST 2017> (x) orb -up</font></div><div><font color="#000000">Tue Jan 17 22:16:17 IST 2017> (x) orb -dn</font></div><div><font color="#000000">Tue Jan 17 22:16:17 IST 2017> (x) lapw1 -up -orb</font></div><div><font color="#000000">Tue Jan 17 22:17:26 IST 2017> (x) lapw2 -up -orb</font></div><div><font color="#000000">Tue Jan 17 22:17:44 IST 2017> (x) lcore -up</font></div><div><font color="#000000">Tue Jan 17 22:17:44 IST 2017> (x) lcore -dn</font></div><div><font color="#000000">Tue Jan 17 22:17:45 IST 2017> (x) mixer -orb</font></div></div><div><font color="#000000"><br></font></div></div></div><div><br></div><div>(3) <span style="color:rgb(0,0,0)">I want to know how to change below variable in the job file so that I can run more effectively mpi run</span></div><div><font color="#000000"><br></font></div><div><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap"># the following number / 4 = number of nodes
#$ -pe mpich 32
set mpijob=1 ??
set jobs_per_node=4 ??
#### the definition above requests 32 cores and we have 4 cores /node.
#### We request only k-point parallel, thus mpijob=1
#### the resulting machines names are in $TMPDIR/machines
setenv OMP_NUM_THREADS 1 ???????</pre><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap"><br></pre><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap">(4) The job with 32 core and with 64 core (with "set mpijob=2") taking ~equal time for scf cycles.</pre><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap"><br></pre><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap"><br></pre><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap">The other compilers options set as:</pre><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap"><br></pre><pre style="word-wrap:break-word"><font color="#000000"><span style="white-space:pre-wrap"> Recommended options for system linuxifc are:
RP_LIB(SCALAPACK+PBLAS): -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 $(R_LIBS)
FPOPT(par.comp.options): -O1 -FR -mp1 -w -prec_div -pc80 -pad -ip -DINTEL_VML -traceback -assume buffered_io -I$(MKLROOT)/include
MPIRUN command : mpirun -np _NP_ -machinefile _HOSTS_ _EXEC_
Current settings:
FFTW_LIB + FFTW_OPT : -lfftw3_mpi -lfftw3 -L/usr/local/lib + -DFFTW3 -I/usr/local/include (already set)
ELPA_LIB + ELPA_OPT : + (already set)
RP RP_LIB(SCALAPACK+PBLAS): -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 $(R_LIBS)
FP FPOPT(par.comp.options): -O1 -FR -mp1 -w -prec_div -pc80 -pad -ip -DINTEL_VML -traceback -assume buffered_io -I$(MKLROOT)/include
MP MPIRUN command : mpirun -np _NP_ -machinefile _HOSTS_ _EXEC_
CN CORES_PER_NODE : 16<br></span></font></pre><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap"><br></pre><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap">For any other supporting information please let me know.</pre><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap"><br></pre><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap">Sincerely</pre><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap">Bhamu</pre></div><div>
</div></div>