[Wien] parallel compilation
Peter Blaha
pblaha at theochem.tuwien.ac.at
Tue Sep 15 17:03:14 CEST 2009
The lapack/blas calls of the mkl are parallelized. This means, that the largest part
of lapw1 /HNS and diagonalization) can be parallelized without any effort.
One can "activiate/deactivate" it using
setenv OMP_NUM_THREAD 2 or 1 (csh syntax, for bash use diferent).
After "userconfig this is even "commented" in your .bashrc/.cshrc file.
Uncomment it and try it out.
Jonas Baltrusaitis schrieb:
> Peter,
>
> are you implying that it is possible to run one job on 2-4 cores in 'parallel' without any mpi and granularization? Could you elaborate on that?
>
> Jonas
>
> --- On Tue, 9/15/09, Peter Blaha <pblaha at theochem.tuwien.ac.at> wrote:
>
>> From: Peter Blaha <pblaha at theochem.tuwien.ac.at>
>> Subject: Re: [Wien] parallel compilation
>> To: "A Mailing list for WIEN2k users" <wien at zeus.theochem.tuwien.ac.at>
>> Date: Tuesday, September 15, 2009, 1:33 AM
>>> if you do not select fine
>> grained parallel, siteconfig_lapw does not give you options
>> to tune parallel parameters. does that mean that it is
>> not compiling parallel at all? if i do not do fine
>> tuning, i can get a complete compilation.
>>
>> If one does NOT select "fine grain parallel"; siteconfig
>> will not produce any
>> mpi-executables. This is clearly the proper choice
>>
>> i) for beginners
>> ii) on "small" computers, where you may not have
>> installed mpi + fftw ;
>> or have only a slow network between them.
>> iii) when you intend to limit your unitcells to about 50
>> atoms/cell.
>>
>> (Still k-point parallelization is possible, and this is
>> much more efficient
>> than mpi for small systems).
>>
>> If you select fine grain parallelization, also the
>> mpi-executables will be
>> produced. But you need:
>>
>> i) a properly installed mpi (+ mpi-fortran compiler, which
>> is usually "created"
>> when mpi is installed. (Usually MPF is
>> NOT ifort, but mpif90 !!)
>> Note: mpi2 needs some small code changes
>> in lapw0.F
>> ii) a corresponding Scalapack+Blacs (also suitable for your
>> compiler)
>> (usually it comes together with the mkl)
>> iii) FFTW routines (compiled with the mpi option.
>> iv) you need to properly specify the "installation place"
>> (-Lxxxx) and the
>> name of the libraries (-lxxxx). Also make
>> sure you understand 32-bit vs
>> 64bit ("emt") issues.
>>
>> In addition, this code is useful ONLY if you have unitcells
>> with more than
>> about 50 atoms !!!! and you have hardware with fast network
>> (Infiniband),
>> i.e. Gbit ethernet may work, but will be slow.
>>
>> On a SINGLE Quadcore-computer, mpi-parallelization is
>> probably NOT worth the effort
>> (use OMP_NUM_THREAD 2 (or 4))
>>
>>
>> linuxif9:MPF:ifort ????
>> unlikely! (see above)
>>
>>
>>
>>> # Linux PC system with IFC 10 compiler + mkl 10 (-ip
>> is broken; -static does not give traceback-lines)
>>> linuxif9:FC:ifortlinuxif9:MPF:ifort
>>> linuxif9:CC:cclinuxif9:FOPT:-FR -mp1 -w
>> -prec_div -pc80 -pad -align -DINTEL_VML -traceback -O
>>> 3 -xW
>>> linuxif9:FPOPT:$(FOPT) -FR -mp1 -w -prec_div -pc80
>> -pad -align -DINTEL_VML -traceback
>> -I/opt/mpich2/includelinuxif9:LDFLAGS: $(FOPT)
>> -L/opt/intel/mkl/10.2.1.017/lib/em64t <http://10.2.1.017/lib/em64t> -pthread -i-static
>>> linuxif9:R_LIBS:-L/opt/intel/mkl/10.2.1.017/lib/em64t
>> <http://10.2.1.017/lib/em64t> -lmkl_lapack -lmkl
>> -liomp5
>>> -lguide -lmkl_core
>>> linuxif9:DPARALLEL:'-DParallel'
>>> linuxif9:RP_LIBS:
>> -L/opt/intel/mkl/10.2.1.017/lib/em64t <http://10.2.1.017/lib/em64t> -lmkl_lapack
>> -lmkl_intel
>>> _lp64 -lmkl_scalapack_lp64 -lmkl_blacs_lp64
>> -lmkl_sequential -lmkl_intel_ilp64 -
>>> lmkl_scalapack_ilp64 -L/opt/fftw-2.1.5/lib/
>> -lfftw_mpi -lfftw -L/opt/mpich2/lib
>>> -lmpich -lfmpich
>>
>> linuxif9:MPIRUN:mpiexec _EXEC_
>>> i am running on a xeon, CentOS5 machine. intel
>> noncommercial.
>>> Any information would be appreciated.
>>>
>>> Thank you,
>>> JD
>>>
>>> On Mon, Sep 14, 2009 at 10:31 AM, Jeff DeReus <jdereus at gmail.com
>> <mailto:jdereus at gmail.com>>
>> wrote:
>>> fftw was compiled into
>> /opt/fftw-2.1.5 with flags
>>> $ ./configure
>> --prefix=/opt/fftw-2.1.5/ --enable-mpi --enable-threads
>>> Thank you,
>>> JD
>>>
>>>
>>> On Mon, Sep 14, 2009 at 10:27
>> AM, Laurence Marks
>>> <L-marks at northwestern.edu
>> <mailto:L-marks at northwestern.edu>>
>> wrote:
>>> It looks like
>> you did not compile fftw (or it is somewhere else).
>>> 2009/9/14 Jeff
>> DeReus <jdereus at gmail.com
>>> <mailto:jdereus at gmail.com>>:
>>> > Hello
>> again. i am having some issues compiling lapw0/1/2
>>> modules in
>>> >
>> wein2k_09. i am running on a CentOS 5.3 box.
>> intel
>>> non-commercial
>>> > compilers and
>> mkl.
>>> >
>>> >
>>> >
>>> > here are my
>> current parallel settings from siteconfig.
>>> >
>>> >
>> Current settings:
>>> >
>> RP RP_LIB(SCALAPACK+PBLAS):
>>>
>> -L/opt/intel/mkl/10.2.1.017/lib/em64t
>> <http://10.2.1.017/lib/em64t>
>>> > -lmkl_lapack
>> -lmkl_intel_lp64 -lmkl_scalapack_lp64
>>>
>> -lmkl_blacs_lp64
>>> >
>> -lmkl_sequential -L/opt/fftw-2.1.5/lib/ -lfftw_mpi -lfftw
>>>
>> -L/opt/mpich2/lib
>>> > -lmpich
>>> >
>> FP FPOPT(par.comp.options): $(FOPT) -FR -mp1
>> -w
>>> -prec_div -pc80
>> -pad
>>> > -align
>> -DINTEL_VML -traceback -I/opt/mpich2/include
>>> >
>> -I/opt/fftw-2.1.5/fortran
>>> >
>> MP MPIRUN commando :
>> mpiexec _EXEC_
>>> >
>>> > all other
>> modules compile correctly. if i do not enable
>> parallel
>>> > functionality,
>> the compilation completes with no errors.
>>> >
>>> > when compiling
>> /opt/wien2k/SRC_lapw0 it ends with these
>>> errors which i
>> have
>>> > not been able
>> to track down.
>>> >
>>> > fftw_para.o: In
>> function `exec_fftw_para_':
>>> >
>> fftw_para.F:(.text+0x77): undefined reference to
>>>
>> `fftwnd_f77_mpi_'
>>> >
>> fftw_para.F:(.text+0xb2): undefined reference to
>>>
>> `fftwnd_f77_mpi_'
>>> > fftw_para.o: In
>> function `init_fftw_para_':
>>> >
>> fftw_para.F:(.text+0x101): undefined reference to
>>> >
>> `fftw3d_f77_mpi_create_plan_'
>>> >
>> fftw_para.F:(.text+0x129): undefined reference to
>>> >
>> `fftw3d_f77_mpi_create_plan_'
>>> >
>> fftw_para.F:(.text+0x14d): undefined reference to
>>> >
>> `fftwnd_f77_mpi_local_sizes_'
>>> > make[1]: ***
>> [lapw0_mpi] Error 1
>>> > make[1]:
>> Leaving directory `/opt/wien2k/SRC_lapw0'
>>> > make: ***
>> [para] Error 2
>>> >
>>> > however,
>> /opt/wien2k/SRC_lapw1/2 end with these errors.
>>> >
>>> > <snip>
>>> >
>> /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>>> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>:
>> undefined
>>> > reference to
>> `mkl_pdett__d_forward_trig_transform'
>>> >
>> /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>>> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>:
>> undefined
>>> > reference to
>> `mkl_trans_mkl_comatcopy'
>>> >
>> /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>>> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>:
>> undefined
>>> > reference to
>> `mkl_pdett_s_init_trig_transform'
>>> >
>> /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>>> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>:
>> undefined
>>> > reference to
>> `mkl_pdepl_s_commit_helmholtz_3d'
>>> >
>> /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>>> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>:
>> undefined
>>> > reference to
>> `mkl_pdepl_d_commit_helmholtz_2d'
>>> >
>> /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>>> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>:
>> undefined
>>> > reference to
>> `mkl_pdett__d_commit_trig_transform'
>>> >
>> /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>>> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>:
>> undefined
>>> > reference to
>> `mkl_trans_mkl_comatadd'
>>> >
>> /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>>> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>:
>> undefined
>>> > reference to
>> `mkl_serv_getversionstring_c'
>>> >
>> /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>>> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>:
>> undefined
>>> > reference to
>> `mkl_spblas_lp64_mkl_dcsrsky'
>>> >
>> /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>>> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>:
>> undefined
>>> > reference to
>> `mkl_pdett__d_init_trig_transform'
>>> > make[1]: ***
>> [lapw2c_mpi] Error 1
>>> > make[1]:
>> Leaving directory `/opt/wien2k/SRC_lapw2'
>>> > make: *** [cp]
>> Error 2
>>> >
>>> >
>>> > i can find no
>> mention of the fftw3d errors anywhere.
>>> >
>>> > my complete
>> appropriate section of siteconfig
>>> >
>>> > # Linux PC
>> system with IFC 10 compiler + mkl 10 (-ip is
>>> broken; -static
>> does
>>> > not give
>> traceback-lines)
>>> >
>> linuxif9:FC:ifort
>>> >
>> linuxif9:MPF:ifort
>>> > linuxif9:CC:cc
>>> >
>> linuxif9:FOPT:-FR -mp1 -w -prec_div -pc80 -pad -align
>>> -DINTEL_VML
>>> > -traceback -O3
>> -xW
>>> >
>> linuxif9:FPOPT:$(FOPT) -FR -mp1 -w -prec_div -pc80 -pad
>>> -align
>> -DINTEL_VML
>>> > -traceback
>> -I/opt/mpich2/include
>>> >
>> linuxif9:LDFLAGS: $(FOPT)
>>>
>> -L/opt/intel/mkl/10.2.1.017/lib/em64t
>>> <http://10.2.1.017/lib/em64t> -pthread
>>> > -i-static
>>> >
>> linuxif9:R_LIBS:-L/opt/intel/mkl/10.2.1.017/lib/em64t
>>> <http://10.2.1.017/lib/em64t> -lmkl_lapack -lmkl
>>> > -liomp5 -lguide
>> -lmkl_core
>>> >
>> linuxif9:DPARALLEL:'-DParallel'
>>> >
>> linuxif9:RP_LIBS: -L/opt/intel/mkl/10.2.1.017/lib/em64t
>>> <http://10.2.1.017/lib/em64t> -lmkl_lapack
>>> >
>> -lmkl_intel_lp64 -lmkl_scalapack_lp64 -lmkl_blacs_lp64
>>>
>> -lmkl_sequential
>>> >
>> -L/opt/fftw-2.1.5/lib/ -lfftw_mpi -lfftw
>>> >
>> linuxif9:MPIRUN:mpiexec _EXEC_
>>> >
>>> >
>>> > any assistance
>> in this matter would be appreciated.
>>> >
>>> > Thank you,
>>> > JD
>>> >
>>> >
>> _______________________________________________
>>> > Wien mailing
>> list
>>> > Wien at zeus.theochem.tuwien.ac.at
>>> <mailto:Wien at zeus.theochem.tuwien.ac.at>
>>> > http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Laurence Marks
>>> Department of
>> Materials Science and Engineering
>>> MSE Rm 2036 Cook
>> Hall
>>> 2220 N Campus
>> Drive
>>> Northwestern
>> University
>>> Evanston, IL
>> 60208, USA
>>> Tel: (847)
>> 491-3996 Fax: (847) 491-7820
>>> email: L-marks
>> at northwestern dot edu
>>> Web:
>> www.numis.northwestern.edu <http://www.numis.northwestern.edu>
>>> Chair,
>> Commission on Electron Crystallography of IUCR
>>>
>> www.numis.northwestern.edu/ <http://www.numis.northwestern.edu/>
>>> Electron
>> crystallography is the branch of science that uses electron
>>> scattering and
>> imaging to study the structure of matter.
>>>
>> _______________________________________________
>>> Wien mailing
>> list
>>> Wien at zeus.theochem.tuwien.ac.at
>>> <mailto:Wien at zeus.theochem.tuwien.ac.at>
>>> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
>>>
>>>
>>>
>>>
>>>
>> ------------------------------------------------------------------------
>>> _______________________________________________
>>> Wien mailing list
>>> Wien at zeus.theochem.tuwien.ac.at
>>> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
>> --
>>
>>
>> P.Blaha
>> --------------------------------------------------------------------------
>> Peter BLAHA, Inst.f. Materials Chemistry, TU Vienna, A-1060
>> Vienna
>> Phone: +43-1-58801-15671
>> FAX: +43-1-58801-15698
>> Email: blaha at theochem.tuwien.ac.at
>> WWW: http://info.tuwien.ac.at/theochem/
>> --------------------------------------------------------------------------
>> _______________________________________________
>> Wien mailing list
>> Wien at zeus.theochem.tuwien.ac.at
>> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
>>
>
>
>
> _______________________________________________
> Wien mailing list
> Wien at zeus.theochem.tuwien.ac.at
> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
--
P.Blaha
--------------------------------------------------------------------------
Peter BLAHA, Inst.f. Materials Chemistry, TU Vienna, A-1060 Vienna
Phone: +43-1-58801-15671 FAX: +43-1-58801-15698
Email: blaha at theochem.tuwien.ac.at WWW: http://info.tuwien.ac.at/theochem/
--------------------------------------------------------------------------
More information about the Wien
mailing list