[Wien] Compiling lapw1_mpi with HP-mpi and MKL
E B Lombardi
lombaeb at science.unisa.ac.za
Fri Jul 10 10:48:55 CEST 2009
Thank you for the details.
Regarding your version of Wien2k: which version are you running (from
the directory names it seems v8)?
(If there were any other responses on the W2k mailing-list, I may have
missed them while the mailing-list was down).
Thanks
Enrico
Oleg Rubel wrote:
> My working configuration is: mvapich2-1.2/Intel fortran 11/MKL 10. I use SGE for job submission that takes care of booting and shutting down MPD. My experience with IntelMPI was not good. (I cannot say anything about m(va)pich.) In order to make Wien2k work with m(va)pich2 on 64-bit machines, some minor modification of lapw0 is necessary as reported in the maillist (http://zeus.theochem.tuwien.ac.at/pipermail/wien/2009-March/012271.html). Please find below some details. I am ready to share additional information, if necessary.
>
> I hope this will help.
>
> Best regards,
>
> Oleg
>
> --
> Oleg Rubel, PhD
> Scientist
> Thunder Bay Regional Research Institute
> 290 Munro St
> Thunder Bay, ON
> P7A 7T1, Canada
> Phone: +1-807-7663350
> Fax: +1-807-3441948
> E-mail: rubelo at tbh.net
> Homepage: http://www.tbrri.com/~orubel/
>
>
> In submission script it is important to have that:
> ### This is important to prevent CPU affinity and have 100% CPU
> ### ===========================================================
> setenv MV2_ENABLE_AFFINITY 0
>
> Further details of the configuration:
> [oleg at feynman mvapich2-1.2]$ cat /etc/*release*
> CentOS release 5.2 (Final)
> [oleg at feynman mvapich2-1.2]$ cat /proc/cpuinfo
> processor : 0
> vendor_id : GenuineIntel
> cpu family : 6
> model : 23
> model name : Intel(R) Xeon(R) CPU E5462 @ 2.80GHz
> stepping : 6
> cpu MHz : 2793.004
> cache size : 6144 KB
> physical id : 0
> siblings : 4
> core id : 0
> cpu cores : 4
> fpu : yes
> fpu_exception : yes
> cpuid level : 10
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr lahf_lm
> bogomips : 5589.78
> clflush size : 64
> cache_alignment : 64
> address sizes : 38 bits physical, 48 bits virtual
> power management:
> ...
> [oleg at feynman WIEN2k_v08.mkl_10_IB_MVAPICH2]$ cat VERSION
> WIEN2k_08.3 (Release 18/9/2008)
> [oleg at feynman WIEN2k_v08.mkl_10_IB_MVAPICH2]$ cat OPTIONS
> current:FOPT:-FR -mp1 -w -prec_div -pc80 -pad -align -DINTEL_VML
> current:FPOPT:$(FOPT)
> current:LDFLAGS:$(FOPT) -L/act/intel/mkl/10.1.0.015/lib/em64t -i-static
> current:DPARALLEL:'-DParallel'
> current:R_LIBS:-Bstatic -lmkl_lapack -lmkl_em64t -lguide -Bdynamic -lpthread
> current:RP_LIBS:-lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -lmkl_core -lmkl_intel_lp64 -lmkl_solver_lp64_sequential -lmkl_em64t -lmkl_sequential -lmkl_intel_thread -lmkl_lapack -lmkl_cdft -lmkl_cdft_core -lmkl_solver_lp64 -lguide -lpthread
> current:MPIRUN:mpiexec -machinefile _HOSTS_ -n _NP_ _EXEC_
> [oleg at feynman WIEN2k_v08.mkl_10_IB_MVAPICH2]$ cat parallel_options
> setenv USE_REMOTE 0
> setenv WIEN_GRANULARITY 1
> setenv WIEN_MPIRUN "mpiexec -machinefile _HOSTS_ -n _NP_ _EXEC_"
> [oleg at feynman ~]$ ifort -v
> Version 11.0
> [oleg at feynman ~]$ env
> MKLROOT=/act/intel/mkl/10.1.0.015
> MODULE_VERSION_STACK=3.2.6
> MANPATH=/act/intel/mkl/10.1.0.015/man:/act/intel/itac/7.2.0.011/man:/act/intel/fc/11.0.074/man:/act/intel/cc/11.0.074/man:/usr/man:/usr/share/man:/usr/local/man:/usr/local/share/man:/usr/X11R6/man:/act/Modules/3.2.6/man
> HOSTNAME=feynman.tbrri.com
> INTEL_LICENSE_FILE=/home/act/intel/licenses:/home/act/intel/licenses:/act/intel/fc/11.0.074/licenses:/opt/intel/licenses:/home/oleg/intel/licenses:/act/intel/cc/11.0.074/licenses:/opt/intel/licenses:/home/oleg/intel/licenses
> TERM=vt100
> SHELL=/bin/bash
> XCRYSDEN_SCRATCH=/gtmp/xcrysden_scratch
> HISTSIZE=1000
> SSH_CLIENT=10.67.195.1 1147 22
> LIBRARY_PATH=/act/intel/mkl/10.1.0.015/lib/em64t
> SGE_CELL=default
> FPATH=/act/intel/mkl/10.1.0.015/include
> OCTAVE_EXEC_PATH=/act/WIEN2k_v08.mkl_10_IB_MVAPICH2:/act/WIEN2k_v08.mkl_10_IB_MVAPICH2/SRC_structeditor/bin
> SSH_TTY=/dev/pts/6
> XCRYSDEN_TOPDIR=/act/XCrySDen-1.5.17-bin-semishared
> USER=oleg
> LD_LIBRARY_PATH=/act/intel/impi/3.2.0.011/lib64:/act/intel/mkl/10.1.0.015/lib/em64t:/act/intel/itac/7.2.0.011/itac/slib_impi3:/act/intel/fc/11.0.074/lib/intel64:/act/intel/cc/11.0.074/lib/intel64
> LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35:
> CPATH=/act/intel/mkl/10.1.0.015/include
> OCTAVE_PATH=/act/WIEN2k_v08.mkl_10_IB_MVAPICH2/SRC_structeditor/bin
> NLSPATH=/act/intel/mkl/10.1.0.015/lib/em64t/locale/%l_%t/%N:/act/intel/fc/11.0.074/lib/intel64/locale/%l_%t/%N:/act/intel/cc/11.0.074/idb/intel64/locale/%l_%t/%N:/act/intel/cc/11.0.074/lib/intel64/locale/%l_%t/%N
> VT_ADD_LIBS=-ldwarf -lelf -lvtunwind -lnsl -lm -ldl -lpthread
> MODULE_VERSION=3.2.6
> MAIL=/var/spool/mail/oleg
> PATH=/act/XCrySDen-1.5.17-bin-semishared:/act/WIEN2k_v08.mkl_10_IB_MVAPICH2:/act/WIEN2k_v08.mkl_10_IB_MVAPICH2/SRC_structeditor/bin:/act/intel/ictce/3.2.0.020/bin:/act/intel/impi/3.2.0.011/bin64:/act/intel/itac/7.2.0.011/bin:/usr/kerberos/bin:/act/intel/fc/11.0.074/bin/intel64:/act/intel/cc/11.0.074/bin/intel64:/act/ge/bin/lx24-amd64:/usr/local/bin:/bin:/usr/bin:/act/bin:/act/bin:/home/oleg/bin
> STRUCTEDIT_PATH=/act/WIEN2k_v08.mkl_10_IB_MVAPICH2/SRC_structeditor/bin
> INPUTRC=/etc/inputrc
> PWD=/home/oleg
> _LMFILES_=/act/Modules/3.2.6/modulefiles/ict:/act/Modules/3.2.6/modulefiles/wien2k_mvapich2
> EDITOR=vi
> IDB_HOME=/act/intel/cc/11.0.074/bin/intel64
> LANG=en_US.UTF-8
> MODULEPATH=/act/Modules/versions:/act/Modules/$MODULE_VERSION/modulefiles:/act/Modules/modulefiles:
> VT_LIB_DIR=/act/intel/itac/7.2.0.011/itac/lib_impi3
> LOADEDMODULES=ict:wien2k_mvapich2
> SGE_ROOT=/act/ge
> VT_ROOT=/act/intel/itac/7.2.0.011
> SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
> SHLVL=1
> HOME=/home/oleg
> VT_SLIB_DIR=/act/intel/itac/7.2.0.011/itac/slib_impi3
> LOGNAME=oleg
> CVS_RSH=ssh
> CLASSPATH=/act/intel/itac/7.2.0.011/itac/lib_impi3
> SSH_CONNECTION=10.67.195.1 1147 10.60.196.1 22
> MODULESHOME=/act/Modules/3.2.6
> OMP_NUM_THREADS=1
> LESSOPEN=|/usr/bin/lesspipe.sh %s
> WIENROOT=/act/WIEN2k_v08.mkl_10_IB_MVAPICH2
> SGE_CLUSTER_NAME=feynman
> INCLUDE=/act/intel/mkl/10.1.0.015/include
> G_BROKEN_FILENAMES=1
> _=/bin/env
>
>
>>>> "Lombardi, Enrico" <Lombaeb at unisa.ac.za> 07/07/09 10:03 AM >>>
>>>>
> This message (and attachments) is subject to restrictions and a disclaimer. Please refer to http://www.unisa.ac.za/disclaimer for full details.
> ________________________________
>
> Dear Wien2k users and authors
>
> We are trying to compile mpi-parallel Wien2k lapw1/2 on an infiniband system, but have not been successful up to now.
>
> We would appreciate an indication of which combinations of mpi-library, math-library and compiler are known to work on infini-band systems? Also what scaling has been achieved on such systems up to now?
>
> Currently, we are compiling using different scenarios:
>
> 1. HP-MPI v2.3.1, Intel Fortran v 11.0 and MKL : In this case the code compiles without error messages, but lapw1 crashes immediately with numerous segfaults.
>
> 2. Still using HP-MPI, with Intel Fortran v11.0, but with selfcompiled ScaLAPACK+BLAS in addition to the Intel MKL, this also compiles smoothly. However lapw1_mpi runtime behaviour depends on how the parallelization is done [mix of mpi+k-parallelization], with some cases resulting in seeming smooth runs, but crashes in lapw2: dnlapw2_XX.error files containing 'l2main' - QTL-B.GT.15., Ghostbands, check scf files". while other combinations of k-point vs mpi-parallelization result in hanging lapw1_mpi jobs which never complete (0% CPU usage, which later segfault).
>
> Note that 'serial' Wien2k (k-point parallelization) always works smoothly.
>
> It would be appreciated if we could obtain known working link/compile options for mpi-parallel lapwX on infiniband systems:
> 1. Which MPI libraries were used?
> 2. Which ScaLAPACK/BLAS, and version?
> 3. Which Compiler and version?
> 4. Linking options and mpirun options?
>
> Please let me know if there are any additional details which are needed.
>
> Any assistance would be appreciated.
>
> Thank you
> Regards
> Enrico Lombardi
>
> NOTES ON INPUT:
> In all cases the tests are based on the standard mpi-parallel benchmark, but increasing the number of k-points to match number of nodes (and first initializing the calculation in the usual way to be able to complete SCF cycles, not just lapw1).
>
> .machines files used:
> K-point parallelization only:
> 1:node1
> 1:node1
> ...
> 1:node2
> 1:node2
> ...
>
> mpi-parallelization only:
> 1:node1:8 node2:8 node3:8 node4:8 .....
>
> mixture of mpi and k-point parallelization:
> 1:node1:8 node2:8 node3:8 .....
> 1:node9:8 node10:8 node11:8 ....
> ....
>
> --
> Dr E B Lombardi
> Physics Department
> University of South Africa
> P.O. Box 392
> UNISA 0003
> Pretoria
> South Africa
>
> Tel: 012 429 8654 / 8027
> Fax: 012 429 3643
> E-mail: lombaeb at unisa.ac.za<mailto:lombaeb at science.unisa.ac.za>
>
>
> _______________________________________________
> Wien mailing list
> Wien at zeus.theochem.tuwien.ac.at
> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
>
>
--
Dr E B Lombardi
Physics Department
University of South Africa
P.O. Box 392
UNISA 0003
Pretoria
South Africa
Tel: +27 (0)12 429 8654 / 8027
Fax: +27 (0)12 429 3643
E-mail: lombaeb at science.unisa.ac.za
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://zeus.theochem.tuwien.ac.at/pipermail/wien/attachments/20090710/3a7c0958/attachment.htm>
More information about the Wien
mailing list