[Wien] parallel compilation
Peter Blaha
pblaha at theochem.tuwien.ac.at
Tue Sep 15 17:06:32 CEST 2009
In lapw0.F you have to replace all
CALL MPI_ADDRESS(efgb(1)%v20, efgb_address(2), ierr)
by
CALL MPI_GET_ADDRESS(efgb(1)%v20, efgb_address(2), ierr)
Intels ifort is also my recommendation.
Check out our faq page on wien2k.at for scripts for queing systems.
Of course, you will have to adapt it for your scheduler.
Jeff DeReus schrieb:
> Peter,
>
> i am curious as to what code changes would need to be made for mpich2.
> especially as i am using the mpich2 libraries for compilation.
>
> are there issues with the ifort compiler? it was my assumption that if
> i was using the intel mkl then the intel fortran compiler would be
> preferred.
>
> this is being run on a 64-bit 64 node cluster. i am interested to know
> if anyone has had success running with a torque/maui scheduler setup and
> threads. as threads and processes are different and i use the OSC
> mpiexec (as opposed to mpirun), as it talks with torque better, have
> issues been run into in that matter? i have personally never tried to
> implement a thread mechanism on a cluster. i could see some issues with
> resource allocation but if anyone has run into it, any information would
> be appreciated.
>
> Thank you,
> JD
>
>
> On Tue, Sep 15, 2009 at 3:33 AM, Peter Blaha
> <pblaha at theochem.tuwien.ac.at <mailto:pblaha at theochem.tuwien.ac.at>> wrote:
>
> if you do not select fine grained parallel, siteconfig_lapw does
> not give you options to tune parallel parameters. does that
> mean that it is not compiling parallel at all? if i do not do
> fine tuning, i can get a complete compilation.
>
>
> If one does NOT select "fine grain parallel"; siteconfig will not
> produce any
> mpi-executables. This is clearly the proper choice
>
> i) for beginners
> ii) on "small" computers, where you may not have installed mpi +
> fftw ;
> or have only a slow network between them.
> iii) when you intend to limit your unitcells to about 50 atoms/cell.
>
> (Still k-point parallelization is possible, and this is much more
> efficient
> than mpi for small systems).
>
> If you select fine grain parallelization, also the mpi-executables
> will be
> produced. But you need:
>
> i) a properly installed mpi (+ mpi-fortran compiler, which is
> usually "created"
> when mpi is installed. (Usually MPF is NOT ifort, but mpif90 !!)
> Note: mpi2 needs some small code changes in lapw0.F
> ii) a corresponding Scalapack+Blacs (also suitable for your compiler)
> (usually it comes together with the mkl)
> iii) FFTW routines (compiled with the mpi option.
> iv) you need to properly specify the "installation place" (-Lxxxx)
> and the
> name of the libraries (-lxxxx). Also make sure you understand
> 32-bit vs
> 64bit ("emt") issues.
>
> In addition, this code is useful ONLY if you have unitcells with
> more than
> about 50 atoms !!!! and you have hardware with fast network
> (Infiniband),
> i.e. Gbit ethernet may work, but will be slow.
>
> On a SINGLE Quadcore-computer, mpi-parallelization is probably NOT
> worth the effort
> (use OMP_NUM_THREAD 2 (or 4))
>
>
> linuxif9:MPF:ifort ???? unlikely! (see above)
>
>
>
> # Linux PC system with IFC 10 compiler + mkl 10 (-ip is broken;
> -static does not give traceback-lines)
> linuxif9:FC:ifortlinuxif9:MPF:ifort
> linuxif9:CC:cclinuxif9:FOPT:-FR -mp1 -w -prec_div -pc80 -pad
> -align -DINTEL_VML -traceback -O
> 3 -xW
> linuxif9:FPOPT:$(FOPT) -FR -mp1 -w -prec_div -pc80 -pad -align
> -DINTEL_VML -traceback -I/opt/mpich2/includelinuxif9:LDFLAGS:
> $(FOPT) -L/opt/intel/mkl/10.2.1.017/lib/em64t
> <http://10.2.1.017/lib/em64t> <http://10.2.1.017/lib/em64t>
> -pthread -i-static
> linuxif9:R_LIBS:-L/opt/intel/mkl/10.2.1.017/lib/em64t
> <http://10.2.1.017/lib/em64t> <http://10.2.1.017/lib/em64t>
> -lmkl_lapack -lmkl -liomp5
>
> -lguide -lmkl_core
> linuxif9:DPARALLEL:'-DParallel'
> linuxif9:RP_LIBS: -L/opt/intel/mkl/10.2.1.017/lib/em64t
> <http://10.2.1.017/lib/em64t> <http://10.2.1.017/lib/em64t>
> -lmkl_lapack -lmkl_intel
>
> _lp64 -lmkl_scalapack_lp64 -lmkl_blacs_lp64 -lmkl_sequential
> -lmkl_intel_ilp64 -
> lmkl_scalapack_ilp64 -L/opt/fftw-2.1.5/lib/ -lfftw_mpi -lfftw
> -L/opt/mpich2/lib
> -lmpich -lfmpich linuxif9:MPIRUN:mpiexec _EXEC_
>
> i am running on a xeon, CentOS5 machine. intel noncommercial.
>
> Any information would be appreciated.
>
> Thank you,
> JD
>
> On Mon, Sep 14, 2009 at 10:31 AM, Jeff DeReus <jdereus at gmail.com
> <mailto:jdereus at gmail.com> <mailto:jdereus at gmail.com
> <mailto:jdereus at gmail.com>>> wrote:
>
> fftw was compiled into /opt/fftw-2.1.5 with flags
>
> $ ./configure --prefix=/opt/fftw-2.1.5/ --enable-mpi
> --enable-threads
>
> Thank you,
> JD
>
>
> On Mon, Sep 14, 2009 at 10:27 AM, Laurence Marks
> <L-marks at northwestern.edu <mailto:L-marks at northwestern.edu>
> <mailto:L-marks at northwestern.edu
> <mailto:L-marks at northwestern.edu>>> wrote:
>
> It looks like you did not compile fftw (or it is
> somewhere else).
>
> 2009/9/14 Jeff DeReus <jdereus at gmail.com
> <mailto:jdereus at gmail.com>
> <mailto:jdereus at gmail.com <mailto:jdereus at gmail.com>>>:
>
> > Hello again. i am having some issues compiling lapw0/1/2
> modules in
> > wein2k_09. i am running on a CentOS 5.3 box. intel
> non-commercial
> > compilers and mkl.
> >
> >
> >
> > here are my current parallel settings from siteconfig.
> >
> > Current settings:
> > RP RP_LIB(SCALAPACK+PBLAS):
> -L/opt/intel/mkl/10.2.1.017/lib/em64t
> <http://10.2.1.017/lib/em64t> <http://10.2.1.017/lib/em64t>
>
> > -lmkl_lapack -lmkl_intel_lp64 -lmkl_scalapack_lp64
> -lmkl_blacs_lp64
> > -lmkl_sequential -L/opt/fftw-2.1.5/lib/ -lfftw_mpi -lfftw
> -L/opt/mpich2/lib
> > -lmpich
> > FP FPOPT(par.comp.options): $(FOPT) -FR -mp1 -w
> -prec_div -pc80 -pad
> > -align -DINTEL_VML -traceback -I/opt/mpich2/include
> > -I/opt/fftw-2.1.5/fortran
> > MP MPIRUN commando : mpiexec _EXEC_
> >
> > all other modules compile correctly. if i do not
> enable parallel
> > functionality, the compilation completes with no errors.
> >
> > when compiling /opt/wien2k/SRC_lapw0 it ends with these
> errors which i have
> > not been able to track down.
> >
> > fftw_para.o: In function `exec_fftw_para_':
> > fftw_para.F:(.text+0x77): undefined reference to
> `fftwnd_f77_mpi_'
> > fftw_para.F:(.text+0xb2): undefined reference to
> `fftwnd_f77_mpi_'
> > fftw_para.o: In function `init_fftw_para_':
> > fftw_para.F:(.text+0x101): undefined reference to
> > `fftw3d_f77_mpi_create_plan_'
> > fftw_para.F:(.text+0x129): undefined reference to
> > `fftw3d_f77_mpi_create_plan_'
> > fftw_para.F:(.text+0x14d): undefined reference to
> > `fftwnd_f77_mpi_local_sizes_'
> > make[1]: *** [lapw0_mpi] Error 1
> > make[1]: Leaving directory `/opt/wien2k/SRC_lapw0'
> > make: *** [para] Error 2
> >
> > however, /opt/wien2k/SRC_lapw1/2 end with these errors.
> >
> > <snip>
> >
> /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>
> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>
> > reference to `mkl_pdett__d_forward_trig_transform'
> >
> /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>
> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>
> > reference to `mkl_trans_mkl_comatcopy'
> >
> /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>
> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>
> > reference to `mkl_pdett_s_init_trig_transform'
> >
> /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>
> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>
> > reference to `mkl_pdepl_s_commit_helmholtz_3d'
> >
> /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>
> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>
> > reference to `mkl_pdepl_d_commit_helmholtz_2d'
> >
> /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>
> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>
> > reference to `mkl_pdett__d_commit_trig_transform'
> >
> /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>
> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>
> > reference to `mkl_trans_mkl_comatadd'
> >
> /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>
> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>
> > reference to `mkl_serv_getversionstring_c'
> >
> /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>
> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>
> > reference to `mkl_spblas_lp64_mkl_dcsrsky'
> >
> /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>
> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>
> > reference to `mkl_pdett__d_init_trig_transform'
> > make[1]: *** [lapw2c_mpi] Error 1
> > make[1]: Leaving directory `/opt/wien2k/SRC_lapw2'
> > make: *** [cp] Error 2
> >
> >
> > i can find no mention of the fftw3d errors anywhere.
> >
> > my complete appropriate section of siteconfig
> >
> > # Linux PC system with IFC 10 compiler + mkl 10 (-ip is
> broken; -static does
> > not give traceback-lines)
> > linuxif9:FC:ifort
> > linuxif9:MPF:ifort
> > linuxif9:CC:cc
> > linuxif9:FOPT:-FR -mp1 -w -prec_div -pc80 -pad -align
> -DINTEL_VML
> > -traceback -O3 -xW
> > linuxif9:FPOPT:$(FOPT) -FR -mp1 -w -prec_div -pc80 -pad
> -align -DINTEL_VML
> > -traceback -I/opt/mpich2/include
> > linuxif9:LDFLAGS: $(FOPT)
> -L/opt/intel/mkl/10.2.1.017/lib/em64t
> <http://10.2.1.017/lib/em64t>
> <http://10.2.1.017/lib/em64t> -pthread
>
> > -i-static
> > linuxif9:R_LIBS:-L/opt/intel/mkl/10.2.1.017/lib/em64t
> <http://10.2.1.017/lib/em64t>
> <http://10.2.1.017/lib/em64t> -lmkl_lapack -lmkl
>
> > -liomp5 -lguide -lmkl_core
> > linuxif9:DPARALLEL:'-DParallel'
> > linuxif9:RP_LIBS:
> -L/opt/intel/mkl/10.2.1.017/lib/em64t <http://10.2.1.017/lib/em64t>
> <http://10.2.1.017/lib/em64t> -lmkl_lapack
>
> > -lmkl_intel_lp64 -lmkl_scalapack_lp64 -lmkl_blacs_lp64
> -lmkl_sequential
> > -L/opt/fftw-2.1.5/lib/ -lfftw_mpi -lfftw
> > linuxif9:MPIRUN:mpiexec _EXEC_
> >
> >
> > any assistance in this matter would be appreciated.
> >
> > Thank you,
> > JD
> >
> > _______________________________________________
> > Wien mailing list
> > Wien at zeus.theochem.tuwien.ac.at
> <mailto:Wien at zeus.theochem.tuwien.ac.at>
> <mailto:Wien at zeus.theochem.tuwien.ac.at
> <mailto:Wien at zeus.theochem.tuwien.ac.at>>
>
> > http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
> >
> >
>
>
>
> --
> Laurence Marks
> Department of Materials Science and Engineering
> MSE Rm 2036 Cook Hall
> 2220 N Campus Drive
> Northwestern University
> Evanston, IL 60208, USA
> Tel: (847) 491-3996 Fax: (847) 491-7820
> email: L-marks at northwestern dot edu
> Web: www.numis.northwestern.edu
> <http://www.numis.northwestern.edu>
> <http://www.numis.northwestern.edu>
>
> Chair, Commission on Electron Crystallography of IUCR
> www.numis.northwestern.edu/
> <http://www.numis.northwestern.edu/>
> <http://www.numis.northwestern.edu/>
>
> Electron crystallography is the branch of science that
> uses electron
> scattering and imaging to study the structure of matter.
> _______________________________________________
> Wien mailing list
> Wien at zeus.theochem.tuwien.ac.at
> <mailto:Wien at zeus.theochem.tuwien.ac.at>
> <mailto:Wien at zeus.theochem.tuwien.ac.at
> <mailto:Wien at zeus.theochem.tuwien.ac.at>>
>
> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
>
>
>
>
> ------------------------------------------------------------------------
>
>
> _______________________________________________
> Wien mailing list
> Wien at zeus.theochem.tuwien.ac.at
> <mailto:Wien at zeus.theochem.tuwien.ac.at>
> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
>
>
> --
>
> P.Blaha
> --------------------------------------------------------------------------
> Peter BLAHA, Inst.f. Materials Chemistry, TU Vienna, A-1060 Vienna
> Phone: +43-1-58801-15671 FAX: +43-1-58801-15698
> Email: blaha at theochem.tuwien.ac.at
> <mailto:blaha at theochem.tuwien.ac.at> WWW:
> http://info.tuwien.ac.at/theochem/
> --------------------------------------------------------------------------
>
> _______________________________________________
> Wien mailing list
> Wien at zeus.theochem.tuwien.ac.at <mailto:Wien at zeus.theochem.tuwien.ac.at>
> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Wien mailing list
> Wien at zeus.theochem.tuwien.ac.at
> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
--
P.Blaha
--------------------------------------------------------------------------
Peter BLAHA, Inst.f. Materials Chemistry, TU Vienna, A-1060 Vienna
Phone: +43-1-58801-15671 FAX: +43-1-58801-15698
Email: blaha at theochem.tuwien.ac.at WWW: http://info.tuwien.ac.at/theochem/
--------------------------------------------------------------------------
More information about the Wien
mailing list