[Wien] parallel compilation

Jeff DeReus jdereus at gmail.com
Tue Sep 15 15:29:36 CEST 2009


Peter,

i am curious as to what code changes would need to be made for mpich2.
especially as i am using the mpich2 libraries for compilation.

are there issues with the ifort compiler?  it was my assumption that if i
was using the intel mkl then the intel fortran compiler would be preferred.


this is being run on a 64-bit 64 node cluster.  i am interested to know if
anyone has had success running with a torque/maui scheduler setup and
threads.  as threads and processes are different and i use the OSC mpiexec
(as opposed to mpirun), as it talks with torque better, have issues been run
into in that matter?  i have personally never tried to implement a thread
mechanism on a cluster.  i could see some issues with resource allocation
but if anyone has run into it, any information would be appreciated.

Thank you,
JD


On Tue, Sep 15, 2009 at 3:33 AM, Peter Blaha
<pblaha at theochem.tuwien.ac.at>wrote:

> if you do not select fine grained parallel, siteconfig_lapw does not give
>> you options to tune parallel parameters.  does that mean that it is not
>> compiling parallel at all?  if i do not do fine tuning, i can get a complete
>> compilation.
>>
>
> If one does NOT select "fine grain parallel"; siteconfig will not produce
> any
> mpi-executables. This is clearly the proper choice
>
> i) for beginners
> ii) on "small" computers, where you may not have installed   mpi + fftw  ;
>    or have only a slow network between them.
> iii) when you intend to limit your unitcells to about 50 atoms/cell.
>
> (Still k-point parallelization is possible, and this is much more efficient
> than mpi for small systems).
>
> If you select fine grain parallelization, also the mpi-executables will be
> produced. But you need:
>
> i) a properly installed mpi (+ mpi-fortran compiler, which is usually
> "created"
>   when mpi is installed. (Usually MPF is NOT  ifort, but   mpif90  !!)
>   Note: mpi2 needs some small code changes in lapw0.F
> ii) a corresponding Scalapack+Blacs (also suitable for your compiler)
>    (usually it comes together with the mkl)
> iii) FFTW routines (compiled with the mpi option.
> iv) you need to properly specify the "installation place" (-Lxxxx) and the
>    name of the libraries (-lxxxx). Also make sure you understand 32-bit vs
>    64bit ("emt") issues.
>
> In addition, this code is useful ONLY if you have unitcells with more than
> about 50 atoms !!!! and you have hardware with fast network (Infiniband),
> i.e. Gbit ethernet may work, but will be slow.
>
> On a SINGLE Quadcore-computer, mpi-parallelization is probably NOT worth
> the effort
> (use  OMP_NUM_THREAD 2 (or 4))
>
>
> linuxif9:MPF:ifort        ????  unlikely! (see above)
>
>
>
>  # Linux PC system with IFC 10 compiler + mkl 10 (-ip is broken; -static
>> does not give traceback-lines)
>> linuxif9:FC:ifortlinuxif9:MPF:ifort
>> linuxif9:CC:cclinuxif9:FOPT:-FR -mp1 -w -prec_div  -pc80 -pad -align
>> -DINTEL_VML -traceback -O
>> 3 -xW
>> linuxif9:FPOPT:$(FOPT) -FR -mp1 -w -prec_div -pc80 -pad -align -DINTEL_VML
>> -traceback -I/opt/mpich2/includelinuxif9:LDFLAGS:  $(FOPT) -L/opt/intel/mkl/
>> 10.2.1.017/lib/em64t <http://10.2.1.017/lib/em64t> -pthread -i-static
>> linuxif9:R_LIBS:-L/opt/intel/mkl/10.2.1.017/lib/em64t <
>> http://10.2.1.017/lib/em64t> -lmkl_lapack -lmkl -liomp5
>>  -lguide -lmkl_core
>> linuxif9:DPARALLEL:'-DParallel'
>> linuxif9:RP_LIBS: -L/opt/intel/mkl/10.2.1.017/lib/em64t <
>> http://10.2.1.017/lib/em64t> -lmkl_lapack -lmkl_intel
>> _lp64 -lmkl_scalapack_lp64 -lmkl_blacs_lp64 -lmkl_sequential
>> -lmkl_intel_ilp64 -
>> lmkl_scalapack_ilp64  -L/opt/fftw-2.1.5/lib/ -lfftw_mpi -lfftw
>> -L/opt/mpich2/lib
>>  -lmpich -lfmpich                     linuxif9:MPIRUN:mpiexec _EXEC_
>>
>> i am running on a xeon, CentOS5 machine.  intel noncommercial.
>>
>> Any information would be appreciated.
>>
>> Thank you,
>> JD
>>
>> On Mon, Sep 14, 2009 at 10:31 AM, Jeff DeReus <jdereus at gmail.com <mailto:
>> jdereus at gmail.com>> wrote:
>>
>>    fftw was compiled into /opt/fftw-2.1.5 with flags
>>
>>      $ ./configure --prefix=/opt/fftw-2.1.5/ --enable-mpi --enable-threads
>>
>>    Thank you,
>>    JD
>>
>>
>>    On Mon, Sep 14, 2009 at 10:27 AM, Laurence Marks
>>    <L-marks at northwestern.edu <mailto:L-marks at northwestern.edu>> wrote:
>>
>>        It looks like you did not compile fftw (or it is somewhere else).
>>
>>        2009/9/14 Jeff DeReus <jdereus at gmail.com
>>        <mailto:jdereus at gmail.com>>:
>>         > Hello again.  i am having some issues compiling lapw0/1/2
>>        modules in
>>         > wein2k_09.  i am running on a CentOS 5.3 box.  intel
>>        non-commercial
>>         > compilers and mkl.
>>         >
>>         >
>>         >
>>         > here are my current parallel settings from siteconfig.
>>         >
>>         >    Current settings:
>>         >      RP  RP_LIB(SCALAPACK+PBLAS):
>>        -L/opt/intel/mkl/10.2.1.017/lib/em64t <http://10.2.1.017/lib/em64t
>> >
>>
>>         > -lmkl_lapack -lmkl_intel_lp64 -lmkl_scalapack_lp64
>>        -lmkl_blacs_lp64
>>         > -lmkl_sequential -L/opt/fftw-2.1.5/lib/ -lfftw_mpi -lfftw
>>        -L/opt/mpich2/lib
>>         > -lmpich
>>         >      FP  FPOPT(par.comp.options): $(FOPT) -FR -mp1 -w
>>        -prec_div -pc80 -pad
>>         > -align -DINTEL_VML -traceback -I/opt/mpich2/include
>>         > -I/opt/fftw-2.1.5/fortran
>>         >      MP  MPIRUN commando        : mpiexec _EXEC_
>>         >
>>         > all other modules compile correctly.  if i do not enable
>> parallel
>>         > functionality, the compilation completes with no errors.
>>         >
>>         > when compiling /opt/wien2k/SRC_lapw0 it ends with these
>>        errors which i have
>>         > not been able to track down.
>>         >
>>         > fftw_para.o: In function `exec_fftw_para_':
>>         > fftw_para.F:(.text+0x77): undefined reference to
>>        `fftwnd_f77_mpi_'
>>         > fftw_para.F:(.text+0xb2): undefined reference to
>>        `fftwnd_f77_mpi_'
>>         > fftw_para.o: In function `init_fftw_para_':
>>         > fftw_para.F:(.text+0x101): undefined reference to
>>         > `fftw3d_f77_mpi_create_plan_'
>>         > fftw_para.F:(.text+0x129): undefined reference to
>>         > `fftw3d_f77_mpi_create_plan_'
>>         > fftw_para.F:(.text+0x14d): undefined reference to
>>         > `fftwnd_f77_mpi_local_sizes_'
>>         > make[1]: *** [lapw0_mpi] Error 1
>>         > make[1]: Leaving directory `/opt/wien2k/SRC_lapw0'
>>         > make: *** [para] Error 2
>>         >
>>         > however, /opt/wien2k/SRC_lapw1/2 end with these errors.
>>         >
>>         > <snip>
>>         > /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>>        <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>>         > reference to `mkl_pdett__d_forward_trig_transform'
>>         > /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>>        <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>>         > reference to `mkl_trans_mkl_comatcopy'
>>         > /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>>        <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>>         > reference to `mkl_pdett_s_init_trig_transform'
>>         > /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>>        <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>>         > reference to `mkl_pdepl_s_commit_helmholtz_3d'
>>         > /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>>        <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>>         > reference to `mkl_pdepl_d_commit_helmholtz_2d'
>>         > /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>>        <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>>         > reference to `mkl_pdett__d_commit_trig_transform'
>>         > /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>>        <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>>         > reference to `mkl_trans_mkl_comatadd'
>>         > /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>>        <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>>         > reference to `mkl_serv_getversionstring_c'
>>         > /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>>        <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>>         > reference to `mkl_spblas_lp64_mkl_dcsrsky'
>>         > /opt/intel/mkl/10.2.1.017/lib/em64t/libmkl_intel_lp64.so
>>        <http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so>: undefined
>>         > reference to `mkl_pdett__d_init_trig_transform'
>>         > make[1]: *** [lapw2c_mpi] Error 1
>>         > make[1]: Leaving directory `/opt/wien2k/SRC_lapw2'
>>         > make: *** [cp] Error 2
>>         >
>>         >
>>         > i can find no mention of the fftw3d errors anywhere.
>>         >
>>         > my complete appropriate section of siteconfig
>>         >
>>         > # Linux PC system with IFC 10 compiler + mkl 10 (-ip is
>>        broken; -static does
>>         > not give traceback-lines)
>>         > linuxif9:FC:ifort
>>         > linuxif9:MPF:ifort
>>         > linuxif9:CC:cc
>>         > linuxif9:FOPT:-FR -mp1 -w -prec_div  -pc80 -pad -align
>>        -DINTEL_VML
>>         > -traceback -O3 -xW
>>         > linuxif9:FPOPT:$(FOPT) -FR -mp1 -w -prec_div -pc80 -pad
>>        -align -DINTEL_VML
>>         > -traceback -I/opt/mpich2/include
>>         > linuxif9:LDFLAGS:  $(FOPT)
>>        -L/opt/intel/mkl/10.2.1.017/lib/em64t
>>        <http://10.2.1.017/lib/em64t> -pthread
>>         > -i-static
>>         > linuxif9:R_LIBS:-L/opt/intel/mkl/10.2.1.017/lib/em64t
>>        <http://10.2.1.017/lib/em64t> -lmkl_lapack -lmkl
>>         > -liomp5 -lguide -lmkl_core
>>         > linuxif9:DPARALLEL:'-DParallel'
>>         > linuxif9:RP_LIBS: -L/opt/intel/mkl/10.2.1.017/lib/em64t
>>        <http://10.2.1.017/lib/em64t> -lmkl_lapack
>>         > -lmkl_intel_lp64 -lmkl_scalapack_lp64 -lmkl_blacs_lp64
>>        -lmkl_sequential
>>         > -L/opt/fftw-2.1.5/lib/ -lfftw_mpi -lfftw
>>         > linuxif9:MPIRUN:mpiexec _EXEC_
>>         >
>>         >
>>         > any assistance in this matter would be appreciated.
>>         >
>>         > Thank you,
>>         > JD
>>         >
>>         > _______________________________________________
>>         > Wien mailing list
>>         > Wien at zeus.theochem.tuwien.ac.at
>>        <mailto:Wien at zeus.theochem.tuwien.ac.at>
>>         > http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
>>         >
>>         >
>>
>>
>>
>>        --
>>        Laurence Marks
>>        Department of Materials Science and Engineering
>>        MSE Rm 2036 Cook Hall
>>        2220 N Campus Drive
>>        Northwestern University
>>        Evanston, IL 60208, USA
>>        Tel: (847) 491-3996 Fax: (847) 491-7820
>>        email: L-marks at northwestern dot edu
>>        Web: www.numis.northwestern.edu <http://www.numis.northwestern.edu
>> >
>>        Chair, Commission on Electron Crystallography of IUCR
>>        www.numis.northwestern.edu/ <http://www.numis.northwestern.edu/>
>>        Electron crystallography is the branch of science that uses
>> electron
>>        scattering and imaging to study the structure of matter.
>>        _______________________________________________
>>        Wien mailing list
>>        Wien at zeus.theochem.tuwien.ac.at
>>        <mailto:Wien at zeus.theochem.tuwien.ac.at>
>>        http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
>>
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Wien mailing list
>> Wien at zeus.theochem.tuwien.ac.at
>> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
>>
>
> --
>
>                                      P.Blaha
> --------------------------------------------------------------------------
> Peter BLAHA, Inst.f. Materials Chemistry, TU Vienna, A-1060 Vienna
> Phone: +43-1-58801-15671             FAX: +43-1-58801-15698
> Email: blaha at theochem.tuwien.ac.at    WWW:
> http://info.tuwien.ac.at/theochem/
> --------------------------------------------------------------------------
>
> _______________________________________________
> Wien mailing list
> Wien at zeus.theochem.tuwien.ac.at
> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://zeus.theochem.tuwien.ac.at/pipermail/wien/attachments/20090915/47870aa1/attachment.htm>


More information about the Wien mailing list