[Wien] parallel compilation
Jeff DeReus
jdereus at gmail.com
Tue Sep 15 15:29:36 CEST 2009
Peter,
i am curious as to what code changes would need to be made for mpich2.
especially as i am using the mpich2 libraries for compilation.
are there issues with the ifort compiler? it was my assumption that if i
was using the intel mkl then the intel fortran compiler would be preferred.
this is being run on a 64-bit 64 node cluster. i am interested to know if
anyone has had success running with a torque/maui scheduler setup and
threads. as threads and processes are different and i use the OSC mpiexec
(as opposed to mpirun), as it talks with torque better, have issues been run
into in that matter? i have personally never tried to implement a thread
mechanism on a cluster. i could see some issues with resource allocation
but if anyone has run into it, any information would be appreciated.
Thank you,
JD
On Tue, Sep 15, 2009 at 3:33 AM, Peter Blaha
<pblaha at theochem.tuwien.ac.at>wrote:
> if you do not select fine grained parallel, siteconfig_lapw does not give
>> you options to tune parallel parameters. does that mean that it is not
>> compiling parallel at all? if i do not do fine tuning, i can get a complete
>> compilation.
>>
>
> If one does NOT select "fine grain parallel"; siteconfig will not produce
> any
> mpi-executables. This is clearly the proper choice
>
> i) for beginners
> ii) on "small" computers, where you may not have installed mpi + fftw ;
> or have only a slow network between them.
> iii) when you intend to limit your unitcells to about 50 atoms/cell.
>
> (Still k-point parallelization is possible, and this is much more efficient
> than mpi for small systems).
>
> If you select fine grain parallelization, also the mpi-executables will be
> produced. But you need:
>
> i) a properly installed mpi (+ mpi-fortran compiler, which is usually
> "created"
> when mpi is installed. (Usually MPF is NOT ifort, but mpif90 !!)
> Note: mpi2 needs some small code changes in lapw0.F
> ii) a corresponding Scalapack+Blacs (also suitable for your compiler)
> (usually it comes together with the mkl)
> iii) FFTW routines (compiled with the mpi option.
> iv) you need to properly specify the "installation place" (-Lxxxx) and the
> name of the libraries (-lxxxx). Also make sure you understand 32-bit vs
> 64bit ("emt") issues.
>
> In addition, this code is useful ONLY if you have unitcells with more than
> about 50 atoms !!!! and you have hardware with fast network (Infiniband),
> i.e. Gbit ethernet may work, but will be slow.
>
> On a SINGLE Quadcore-computer, mpi-parallelization is probably NOT worth
> the effort
> (use OMP_NUM_THREAD 2 (or 4))
>
>
> linuxif9:MPF:ifort ???? unlikely! (see above)
>
>
>
> # Linux PC system with IFC 10 compiler + mkl 10 (-ip is broken; -static
>> does not give traceback-lines)
>> linuxif9:FC:ifortlinuxif9:MPF:ifort
>> linuxif9:CC:cclinuxif9:FOPT:-FR -mp1 -w -prec_div -pc80 -pad -align
>> -DINTEL_VML -traceback -O
>> 3 -xW
>> linuxif9:FPOPT:$(FOPT) -FR -mp1 -w -prec_div -pc80 -pad -align -DINTEL_VML
>> -traceback -I/opt/mpich2/includelinuxif9:LDFLAGS: $(FOPT) -L/opt/intel/mkl/
>> 10.2.1.017/lib/em64t <http://10.2.1.017/lib/em64t> -pthread -i-static
>> linuxif9:R_LIBS:-L/opt/intel/mkl/10.2.1.017/lib/em64t <
>> http://10.2.1.017/lib/em64t> -lmkl_lapack -lmkl -liomp5
>> -lguide -lmkl_core
>> linuxif9:DPARALLEL:'-DParallel'
>> linuxif9:RP_LIBS: -L/opt/intel/mkl/10.2.1.017/lib/em64t <
>> http://10.2.1.017/lib/em64t> -lmkl_lapack -lmkl_intel
>> _lp64 -lmkl_scalapack_lp64 -lmkl_blacs_lp64 -lmkl_sequential
>> -lmkl_intel_ilp64 -
>> lmkl_scalapack_ilp64 -L/opt/fftw-2.1.5/lib/ -lfftw_mpi -lfftw
>> -L/opt/mpich2/lib
>> -lmpich -lfmpich linuxif9:MPIRUN:mpiexec _EXEC_
>>
>> i am running on a xeon, CentOS5 machine. intel noncommercial.
>>
>> Any information would be appreciated.
>>
>> Thank you,
>> JD
>>
>> On Mon, Sep 14, 2009 at 10:31 AM, Jeff DeReus <jdereus at gmail.com <mailto:
>> jdereus at gmail.com>> wrote:
>>
>> fftw was compiled into /opt/fftw-2.1.5 with flags
>>
>> $ ./configure --prefix=/opt/fftw-2.1.5/ --enable-mpi --enable-threads
>>
>> Thank you,
>> JD
>>
>>
>> On Mon, Sep 14, 2009 at 10:27 AM, Laurence Marks
>> <L-marks at northwestern.edu <mailto:L-marks at northwestern.edu>> wrote:
>>
>> It looks like you did not compile fftw (or it is somewhere else).
>>
>> 2009/9/14 Jeff DeReus <jdereus at gmail.com
>> <mailto:jdereus at gmail.com>>:
>> > Hello again. i am having some issues compiling lapw0/1/2
>> modules in
>> > wein2k_09. i am running on a CentOS 5.3 box. intel
>> non-commercial
>> > compilers and mkl.
>> >
>> >
>> >
>> > here are my current parallel settings from siteconfig.
>> >
>> > Current settings:
>> > RP RP_LIB(SCALAPACK+PBLAS):
>> -L/opt/intel/mkl/10.2.1.017/lib/em64t <http://10.2.1.017/lib/em64t
>> >
>>
>> > -lmkl_lapack -lmkl_intel_lp64 -lmkl_scalapack_lp64
>> -lmkl_blacs_lp64
>> > -lmkl_sequential -L/opt/fftw-2.1.5/lib/ -lfftw_mpi -lfftw
>> -L/opt/mpich2/lib
>> > -lmpich
>> > FP FPOPT(par.comp.options): $(FOPT) -FR -mp1 -w
>> -prec_div -pc80 -pad
>> > -align -DINTEL_VML -traceback -I/opt/mpich2/include
>> > -I/opt/fftw-2.1.5/fortran
>> > MP MPIRUN commando : mpiexec _EXEC_
>> >
>> > all other modules compile correctly. if i do not enable
>> parallel
>> > functionality, the compilation completes with no errors.
>> >
>> > when compiling /opt/wien2k/SRC_lapw0 it ends with these
>> errors which i have
>> > not been able to track down.
>> >
>> > fftw_para.o: In function `exec_fftw_para_':
>> > fftw_para.F:(.text+0x77): undefined reference to
>> `fftwnd_f77_mpi_'
>> > fftw_para.F:(.text+0xb2): undefined reference to
>> `fftwnd_f77_mpi_'
>> > fftw_para.o: In function `init_fftw_para_':
>> > fftw_para.F:(.text+0x101): undefined reference to
>> > `fftw3d_f77_mpi_create_plan_'
>> > fftw_para.F:(.text+0x129): undefined reference to
>> > `fftw3d_f77_mpi_create_plan_'
>> > fftw_para.F:(.text+0x14d): undefined reference to
>> > `fftwnd_f77_mpi_local_sizes_'
>> > make[1]: *** [lapw0_mpi] Error 1
>> > make[1]: Leaving directory `/opt/wien2k/SRC_lapw0'
>> > make: *** [para] Error 2
>> >
>> > however, /opt/wien2k/SRC_lapw1/2 end with these errors.
>> >
>> > <snip>
>> > /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>> > reference to `mkl_pdett__d_forward_trig_transform'
>> > /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>> > reference to `mkl_trans_mkl_comatcopy'
>> > /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>> > reference to `mkl_pdett_s_init_trig_transform'
>> > /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>> > reference to `mkl_pdepl_s_commit_helmholtz_3d'
>> > /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>> > reference to `mkl_pdepl_d_commit_helmholtz_2d'
>> > /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>> > reference to `mkl_pdett__d_commit_trig_transform'
>> > /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>> > reference to `mkl_trans_mkl_comatadd'
>> > /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>> > reference to `mkl_serv_getversionstring_c'
>> > /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>> > reference to `mkl_spblas_lp64_mkl_dcsrsky'
>> > /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>> <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>> > reference to `mkl_pdett__d_init_trig_transform'
>> > make[1]: *** [lapw2c_mpi] Error 1
>> > make[1]: Leaving directory `/opt/wien2k/SRC_lapw2'
>> > make: *** [cp] Error 2
>> >
>> >
>> > i can find no mention of the fftw3d errors anywhere.
>> >
>> > my complete appropriate section of siteconfig
>> >
>> > # Linux PC system with IFC 10 compiler + mkl 10 (-ip is
>> broken; -static does
>> > not give traceback-lines)
>> > linuxif9:FC:ifort
>> > linuxif9:MPF:ifort
>> > linuxif9:CC:cc
>> > linuxif9:FOPT:-FR -mp1 -w -prec_div -pc80 -pad -align
>> -DINTEL_VML
>> > -traceback -O3 -xW
>> > linuxif9:FPOPT:$(FOPT) -FR -mp1 -w -prec_div -pc80 -pad
>> -align -DINTEL_VML
>> > -traceback -I/opt/mpich2/include
>> > linuxif9:LDFLAGS: $(FOPT)
>> -L/opt/intel/mkl/10.2.1.017/lib/em64t
>> <http://10.2.1.017/lib/em64t> -pthread
>> > -i-static
>> > linuxif9:R_LIBS:-L/opt/intel/mkl/10.2.1.017/lib/em64t
>> <http://10.2.1.017/lib/em64t> -lmkl_lapack -lmkl
>> > -liomp5 -lguide -lmkl_core
>> > linuxif9:DPARALLEL:'-DParallel'
>> > linuxif9:RP_LIBS: -L/opt/intel/mkl/10.2.1.017/lib/em64t
>> <http://10.2.1.017/lib/em64t> -lmkl_lapack
>> > -lmkl_intel_lp64 -lmkl_scalapack_lp64 -lmkl_blacs_lp64
>> -lmkl_sequential
>> > -L/opt/fftw-2.1.5/lib/ -lfftw_mpi -lfftw
>> > linuxif9:MPIRUN:mpiexec _EXEC_
>> >
>> >
>> > any assistance in this matter would be appreciated.
>> >
>> > Thank you,
>> > JD
>> >
>> > _______________________________________________
>> > Wien mailing list
>> > Wien at zeus.theochem.tuwien.ac.at
>> <mailto:Wien at zeus.theochem.tuwien.ac.at>
>> > http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
>> >
>> >
>>
>>
>>
>> --
>> Laurence Marks
>> Department of Materials Science and Engineering
>> MSE Rm 2036 Cook Hall
>> 2220 N Campus Drive
>> Northwestern University
>> Evanston, IL 60208, USA
>> Tel: (847) 491-3996 Fax: (847) 491-7820
>> email: L-marks at northwestern dot edu
>> Web: www.numis.northwestern.edu <http://www.numis.northwestern.edu
>> >
>> Chair, Commission on Electron Crystallography of IUCR
>> www.numis.northwestern.edu/ <http://www.numis.northwestern.edu/>
>> Electron crystallography is the branch of science that uses
>> electron
>> scattering and imaging to study the structure of matter.
>> _______________________________________________
>> Wien mailing list
>> Wien at zeus.theochem.tuwien.ac.at
>> <mailto:Wien at zeus.theochem.tuwien.ac.at>
>> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
>>
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Wien mailing list
>> Wien at zeus.theochem.tuwien.ac.at
>> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
>>
>
> --
>
> P.Blaha
> --------------------------------------------------------------------------
> Peter BLAHA, Inst.f. Materials Chemistry, TU Vienna, A-1060 Vienna
> Phone: +43-1-58801-15671 FAX: +43-1-58801-15698
> Email: blaha at theochem.tuwien.ac.at WWW:
> http://info.tuwien.ac.at/theochem/
> --------------------------------------------------------------------------
>
> _______________________________________________
> Wien mailing list
> Wien at zeus.theochem.tuwien.ac.at
> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://zeus.theochem.tuwien.ac.at/pipermail/wien/attachments/20090915/47870aa1/attachment.htm>
More information about the Wien
mailing list