<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="font-family: arial, verdana, sans-serif; font-size: 14px; line-height: 25px; "><div>Dear WIEN2k developers and users,</div><div><br></div><div>We are trying to install WIEN2k 10.1 on a computing cluster and plan to calculate some big system (over 60 atoms/cell) with it. We got no error message during the compilation, and testing with the three examples (Fccni, TiC and TiO2) in serial finished fast and correctly. However we failed in the parallel (k-point and/ or MPI) mode. Therefore, we write here to this email list hoping someone can offer us some help. Below's the details of our system, compilers, libraries, compiler options, linking flags and testing. </div><div><br></div><div><b>1. System :</b> SUSE Linux Enterprise Server 10 (x86_64), Intel Xeon X5355 quad core processors (Intel 64), 2 GB memory per core, DDR 4X InfiniBand, PBS Professional queuing system. </div><div><b><br></b></div><div><b>2. compiler/libraries</b>:<b> <span class="Apple-style-span" style="font-weight: normal; ">ifort and icc </span></b>of Intel 11.1/046, mpiifort<b> </b>of Intel MPI 3.2.0.011, BLAS, LAPACK and scaLAPCK of Intel MKL 10.2, and fftw 2.1.5 (compiled with "--enable-mpi" switch at /home/user/fftw-2.1.5)</div><div>The environment was configured by source in bash_profile:</div><div>source /usr/local/intel/Compiler/11.1/046/bin/ifortvars.sh intel64 #ifort</div><div>source /usr/local/intel/Compiler/11.1/046/mkl/tools/environment/mklvarsem64t.sh #mkl</div><div>source /usr/local/intel/impi/3.2.0.011/bin64/mpivars.sh #mpi</div><div>Their bin, library, and include directory were all sourced in bash_profile as well. </div><div><br></div><div>3. <b>Compiler options</b>: </div><div>For serial:</div><div><div><div> <b>O</b> Compiler options: -FR -mp1 -w -prec_div -pc80 -pad -ip -DINTEL_VML -traceback</div><div><b> L</b> Linker Flags: $(FOPT) -L/opt/intel/Compiler/11.1/046/mkl/lib/em64t -pthread</div><div> <b>P</b> Preprocessor flags '-DParallel'</div><div> <b>R</b> R_LIB (LAPACK+BLAS): -lmkl_lapack -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -openmp -lpthread -lguide</div></div></div><div><br></div><div>For parallel:</div><div><div><b>Shared Memory Architecture</b>: no; </div><div><b>Remote shell</b>: ssh (password-less log-in enabled);</div><div><b>RP</b> -L/usr/local/intel/Compiler/11.1/046/mkl/lib/em64t -lmkl_scalapack_lp64 /usr/local/intel/Compiler/11.1/046/mkl/lib/em64t/libmkl_solver_lp64.a -Wl,--start-group -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64 -Wl,--end-group -openmp -lpthread -L/home/user/fftw-2.1.5/lib -lfftw_mpi -lfftw $(R_LIBS)</div><div> <b>FP</b> FPOPT(par.comp.options): -FR -mp1 -w -prec_div -pc80 -pad -ip -DINTEL_VML -traceback</div><div> <b>MP</b> MPIRUN commando : mpirun -np _NP_ -machinefile _HOSTS_ _EXEC_</div></div><div><br></div><div>Note: We used all WIEN2k recommended options/flags except RP for which we used those from Intel MKL linking Advisor by specifying dynamic, 32-bit (lp64) and multi-threaded etc. We're not sure if these are correct (especially the integers length) and would like to here your suggestion. You may find our processors' specifications are at <a href="http://ark.intel.com/Product.aspx?id=28035">http://ark.intel.com/Product.aspx?id=28035</a> . </div><div><span class="Apple-style-span" style="font-family: arial, sans-serif; line-height: 15px; font-size: small; color: rgb(14, 119, 74); "><br></span></div><div><b>4. Testing</b></div><div><b>4.1 Inputs</b></div><div>We used <i>userconfig_lapw</i> to set the user environment (especially, scratch director is set to be /scratch), and then performed the testing using the Fccni example downloaded from the WIEN2k website. </div><div><br></div><div>We first ran a spin-polarized calculation in serial using the recommended parameters from the User's Guide for the initialization. The calculation finished without problem quickly and the results matched the downloaded outputs well. We then ran <i>save_lapw</i> and <i>clean_lapw</i> so that we can use the same set of input files to test parallelization. We wrote a submission script to create the .machines file and calculate the number of processors allocated ($nprocs) on the fly and start the calculation with: <i>mpirun -np $nprocs runsp_lapw -p -ec 0.0001 -cc 0.0001</i>. We enabled hybrid parallelization (i.e., both k-point and MPI) in this case.</div><div><br></div><div>The .machines file created reads: </div><div><div>1:r1i0n0:8</div><div>1:r1i0n1:8</div><div>lapw0: r1i0n0:8 r1i0n1:8 </div><div>lapw1: r1i0n0:8 r1i0n1:8 </div><div>lapw2: r1i0n0:8 r1i0n1:8 </div><div>granularity:1</div><div>extrafine:1</div></div><div><br></div><div>In this example we were allocated two nodes (r1i0n0 and r1i0n1) by PBS, each with 8 cores (each node is made of two quad-core CPUs which together make 8 cores). The first two lines are for k-point and the next three for MPI (for lapw0, lapw1, lapw2, respectively). </div><div><br></div><div><b>4.2 Outputs</b></div><div>The job was killed within one minute outputting error messages like:</div><div>~ cat aU_SOC.e799326</div><div><div>rm: cannot remove `fccni.vspup': No such file or directory</div><div>rm: cannot remove `fccni.vspdn': No such file or directory</div><div>rm: cannot remove `fccni.vnsup': No such file or directory</div><div>rm: cannot remove `fccni.vnsdn': No such file or directory</div><div>/tmp/pbs.799326.service2/sh.piTkRT: No such file or directory.</div><div>/tmp/pbs.799326.service2/sh.ygkvzW: No such file or directory.</div><div>/tmp/pbs.799326.service2/sh.i4xOi2: No such file or directory.</div><div>mv: cannot stat `.tmp': No such file or directory</div><div>foreach: No match.</div><div>/tmp/pbs.799326.service2/sh.m3zD88: No such file or directory.</div><div>/tmp/pbs.799326.service2/sh.xgo6Fb: No such file or directory.</div><div>/tmp/pbs.799326.service2/sh.zyICya: No such file or directory.</div><div>/tmp/pbs.799326.service2/sh.fI8qUa: No such file or directory.</div><div>/tmp/pbs.799326.service2/sh.cghNSa: No such file or directory.</div><div>foreach: No match.</div><div>mv: cannot stat `.tmp': No such file or directory</div><div>rm: No match.</div><div>rm: cannot remove `fccni.vns': No such file or directory</div><div>rm: cannot remove `fccni.vnsup': No such file or directory</div><div>rm: cannot remove `fccni.vnsdn': No such file or directory</div><div>rm: cannot remove `fccni.vsp': No such file or directory</div><div>rm: cannot remove `fccni.vspdn': No such file or directory</div><div>sed: can't read .machinetmp22: No such file or directory</div><div>rm: cannot remove `.machinetmp': No such file or directory</div><div>machine_i: Subscript out of range.</div><div>cut: .machine0: No such file or directory</div><div>rm: cannot remove `.machinetmp22': No such file or directory</div><div>sed: can't read .machinetmp: No such file or directory</div><div>rm: cannot remove `.machinetmp': No such file or directory</div><div>mv: cannot stat `.tmp': No such file or directory</div><div> LAPW0 END</div><div> LAPW0 END</div><div>@: Expression Syntax.</div></div><div><br></div><div>It seemed that the job stopped when executing LAPW0 because WIEN2k couldn't find/move/delete some files. </div><div><br></div><div>We have tried a couple of different compilations (e.g., using exactly what WIEN2k recommended for <b>RP</b>) but these errors persist. We have also searched the WIEN2k mail list but didn't find any related post. </div><div><br></div><div>Does anyone have any idea on this? Your comments will be highly appreciated! </div><div><br></div><div>Thanks,</div><div>Wei </div></span><div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">-------------------------------------------</div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Computational Materials Group<br>University of Wisconsin-Madison <br>209 MS&E Bldg, 1509 University Ave <br>Madison, WI 53706-1595<br>Office: (608)262-2088 <br>Email: <a href="mailto:wxie4@wisc.edu">wxie4@wisc.edu</a><br>Web: <a href="http://matmodel.engr.wisc.edu/">http://matmodel.engr.wisc.edu/</a></div></span></div></div></div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "></div></span></div></div></div></div></div><br></body></html>