Peter,<br><br>i am curious as to what code changes would need to be made for mpich2. especially as i am using the mpich2 libraries for compilation. <br><br>are there issues with the ifort compiler? it was my assumption that if i was using the intel mkl then the intel fortran compiler would be preferred. <br>
<br>this is being run on a 64-bit 64 node cluster. i am interested to know if anyone has had success running with a torque/maui scheduler setup and threads. as threads and processes are different and i use the OSC mpiexec (as opposed to mpirun), as it talks with torque better, have issues been run into in that matter? i have personally never tried to implement a thread mechanism on a cluster. i could see some issues with resource allocation but if anyone has run into it, any information would be appreciated.<br>
<br>Thank you,<br>JD<br><br><br><div class="gmail_quote">On Tue, Sep 15, 2009 at 3:33 AM, Peter Blaha <span dir="ltr"><<a href="mailto:pblaha@theochem.tuwien.ac.at">pblaha@theochem.tuwien.ac.at</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="im"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
if you do not select fine grained parallel, siteconfig_lapw does not give you options to tune parallel parameters. does that mean that it is not compiling parallel at all? if i do not do fine tuning, i can get a complete compilation. <br>
</blockquote>
<br></div>
If one does NOT select "fine grain parallel"; siteconfig will not produce any<br>
mpi-executables. This is clearly the proper choice<br>
<br>
i) for beginners<br>
ii) on "small" computers, where you may not have installed mpi + fftw ;<br>
or have only a slow network between them.<br>
iii) when you intend to limit your unitcells to about 50 atoms/cell.<br>
<br>
(Still k-point parallelization is possible, and this is much more efficient<br>
than mpi for small systems).<br>
<br>
If you select fine grain parallelization, also the mpi-executables will be<br>
produced. But you need:<br>
<br>
i) a properly installed mpi (+ mpi-fortran compiler, which is usually "created"<br>
when mpi is installed. (Usually MPF is NOT ifort, but mpif90 !!)<br>
Note: mpi2 needs some small code changes in lapw0.F<br>
ii) a corresponding Scalapack+Blacs (also suitable for your compiler)<br>
(usually it comes together with the mkl)<br>
iii) FFTW routines (compiled with the mpi option.<br>
iv) you need to properly specify the "installation place" (-Lxxxx) and the<br>
name of the libraries (-lxxxx). Also make sure you understand 32-bit vs<br>
64bit ("emt") issues.<br>
<br>
In addition, this code is useful ONLY if you have unitcells with more than<br>
about 50 atoms !!!! and you have hardware with fast network (Infiniband),<br>
i.e. Gbit ethernet may work, but will be slow.<br>
<br>
On a SINGLE Quadcore-computer, mpi-parallelization is probably NOT worth the effort<br>
(use OMP_NUM_THREAD 2 (or 4))<br>
<br>
<br>
linuxif9:MPF:ifort ???? unlikely! (see above)<br>
<br>
<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="im">
# Linux PC system with IFC 10 compiler + mkl 10 (-ip is broken; -static does not give traceback-lines)<br>
linuxif9:FC:ifortlinuxif9:MPF:ifort<br>
linuxif9:CC:cclinuxif9:FOPT:-FR -mp1 -w -prec_div -pc80 -pad -align -DINTEL_VML -traceback -O<br>
3 -xW<br></div>
linuxif9:FPOPT:$(FOPT) -FR -mp1 -w -prec_div -pc80 -pad -align -DINTEL_VML -traceback -I/opt/mpich2/includelinuxif9:LDFLAGS: $(FOPT) -L/opt/intel/mkl/<a href="http://10.2.1.017/lib/em64t" target="_blank">10.2.1.017/lib/em64t</a> <<a href="http://10.2.1.017/lib/em64t" target="_blank">http://10.2.1.017/lib/em64t</a>> -pthread -i-static<br>
linuxif9:R_LIBS:-L/opt/intel/mkl/<a href="http://10.2.1.017/lib/em64t" target="_blank">10.2.1.017/lib/em64t</a> <<a href="http://10.2.1.017/lib/em64t" target="_blank">http://10.2.1.017/lib/em64t</a>> -lmkl_lapack -lmkl -liomp5<div class="im">
<br>
-lguide -lmkl_core<br>
linuxif9:DPARALLEL:'-DParallel'<br></div>
linuxif9:RP_LIBS: -L/opt/intel/mkl/<a href="http://10.2.1.017/lib/em64t" target="_blank">10.2.1.017/lib/em64t</a> <<a href="http://10.2.1.017/lib/em64t" target="_blank">http://10.2.1.017/lib/em64t</a>> -lmkl_lapack -lmkl_intel<div class="im">
<br>
_lp64 -lmkl_scalapack_lp64 -lmkl_blacs_lp64 -lmkl_sequential -lmkl_intel_ilp64 -<br>
lmkl_scalapack_ilp64 -L/opt/fftw-2.1.5/lib/ -lfftw_mpi -lfftw -L/opt/mpich2/lib<br>
-lmpich -lfmpich linuxif9:MPIRUN:mpiexec _EXEC_<br>
<br>
i am running on a xeon, CentOS5 machine. intel noncommercial.<br>
<br>
Any information would be appreciated.<br>
<br>
Thank you,<br>
JD<br>
<br></div><div class="im">
On Mon, Sep 14, 2009 at 10:31 AM, Jeff DeReus <<a href="mailto:jdereus@gmail.com" target="_blank">jdereus@gmail.com</a> <mailto:<a href="mailto:jdereus@gmail.com" target="_blank">jdereus@gmail.com</a>>> wrote:<br>
<br>
fftw was compiled into /opt/fftw-2.1.5 with flags<br>
<br>
$ ./configure --prefix=/opt/fftw-2.1.5/ --enable-mpi --enable-threads<br>
<br>
Thank you,<br>
JD<br>
<br>
<br>
On Mon, Sep 14, 2009 at 10:27 AM, Laurence Marks<br></div><div class="im">
<<a href="mailto:L-marks@northwestern.edu" target="_blank">L-marks@northwestern.edu</a> <mailto:<a href="mailto:L-marks@northwestern.edu" target="_blank">L-marks@northwestern.edu</a>>> wrote:<br>
<br>
It looks like you did not compile fftw (or it is somewhere else).<br>
<br>
2009/9/14 Jeff DeReus <<a href="mailto:jdereus@gmail.com" target="_blank">jdereus@gmail.com</a><br></div>
<mailto:<a href="mailto:jdereus@gmail.com" target="_blank">jdereus@gmail.com</a>>>:<div class="im"><br>
> Hello again. i am having some issues compiling lapw0/1/2<br>
modules in<br>
> wein2k_09. i am running on a CentOS 5.3 box. intel<br>
non-commercial<br>
> compilers and mkl.<br>
><br>
><br>
><br>
> here are my current parallel settings from siteconfig.<br>
><br>
> Current settings:<br>
> RP RP_LIB(SCALAPACK+PBLAS):<br></div>
-L/opt/intel/mkl/<a href="http://10.2.1.017/lib/em64t" target="_blank">10.2.1.017/lib/em64t</a> <<a href="http://10.2.1.017/lib/em64t" target="_blank">http://10.2.1.017/lib/em64t</a>><div><div></div><div class="h5">
<br>
> -lmkl_lapack -lmkl_intel_lp64 -lmkl_scalapack_lp64<br>
-lmkl_blacs_lp64<br>
> -lmkl_sequential -L/opt/fftw-2.1.5/lib/ -lfftw_mpi -lfftw<br>
-L/opt/mpich2/lib<br>
> -lmpich<br>
> FP FPOPT(par.comp.options): $(FOPT) -FR -mp1 -w<br>
-prec_div -pc80 -pad<br>
> -align -DINTEL_VML -traceback -I/opt/mpich2/include<br>
> -I/opt/fftw-2.1.5/fortran<br>
> MP MPIRUN commando : mpiexec _EXEC_<br>
><br>
> all other modules compile correctly. if i do not enable parallel<br>
> functionality, the compilation completes with no errors.<br>
><br>
> when compiling /opt/wien2k/SRC_lapw0 it ends with these<br>
errors which i have<br>
> not been able to track down.<br>
><br>
> fftw_para.o: In function `exec_fftw_para_':<br>
> fftw_para.F:(.text+0x77): undefined reference to<br>
`fftwnd_f77_mpi_'<br>
> fftw_para.F:(.text+0xb2): undefined reference to<br>
`fftwnd_f77_mpi_'<br>
> fftw_para.o: In function `init_fftw_para_':<br>
> fftw_para.F:(.text+0x101): undefined reference to<br>
> `fftw3d_f77_mpi_create_plan_'<br>
> fftw_para.F:(.text+0x129): undefined reference to<br>
> `fftw3d_f77_mpi_create_plan_'<br>
> fftw_para.F:(.text+0x14d): undefined reference to<br>
> `fftwnd_f77_mpi_local_sizes_'<br>
> make[1]: *** [lapw0_mpi] Error 1<br>
> make[1]: Leaving directory `/opt/wien2k/SRC_lapw0'<br>
> make: *** [para] Error 2<br>
><br>
> however, /opt/wien2k/SRC_lapw1/2 end with these errors.<br>
><br>
> <snip><br>
> /opt/intel/mkl/<a href="http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so" target="_blank">10.2.1.017/lib/em64t/libmkl_intel_lp64.so</a><br></div></div>
<<a href="http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so" target="_blank">http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so</a>>: undefined<div class="im"><br>
> reference to `mkl_pdett__d_forward_trig_transform'<br>
> /opt/intel/mkl/<a href="http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so" target="_blank">10.2.1.017/lib/em64t/libmkl_intel_lp64.so</a><br></div>
<<a href="http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so" target="_blank">http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so</a>>: undefined<div class="im"><br>
> reference to `mkl_trans_mkl_comatcopy'<br>
> /opt/intel/mkl/<a href="http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so" target="_blank">10.2.1.017/lib/em64t/libmkl_intel_lp64.so</a><br></div>
<<a href="http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so" target="_blank">http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so</a>>: undefined<div class="im"><br>
> reference to `mkl_pdett_s_init_trig_transform'<br>
> /opt/intel/mkl/<a href="http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so" target="_blank">10.2.1.017/lib/em64t/libmkl_intel_lp64.so</a><br></div>
<<a href="http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so" target="_blank">http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so</a>>: undefined<div class="im"><br>
> reference to `mkl_pdepl_s_commit_helmholtz_3d'<br>
> /opt/intel/mkl/<a href="http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so" target="_blank">10.2.1.017/lib/em64t/libmkl_intel_lp64.so</a><br></div>
<<a href="http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so" target="_blank">http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so</a>>: undefined<div class="im"><br>
> reference to `mkl_pdepl_d_commit_helmholtz_2d'<br>
> /opt/intel/mkl/<a href="http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so" target="_blank">10.2.1.017/lib/em64t/libmkl_intel_lp64.so</a><br></div>
<<a href="http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so" target="_blank">http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so</a>>: undefined<div class="im"><br>
> reference to `mkl_pdett__d_commit_trig_transform'<br>
> /opt/intel/mkl/<a href="http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so" target="_blank">10.2.1.017/lib/em64t/libmkl_intel_lp64.so</a><br></div>
<<a href="http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so" target="_blank">http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so</a>>: undefined<div class="im"><br>
> reference to `mkl_trans_mkl_comatadd'<br>
> /opt/intel/mkl/<a href="http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so" target="_blank">10.2.1.017/lib/em64t/libmkl_intel_lp64.so</a><br></div>
<<a href="http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so" target="_blank">http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so</a>>: undefined<div class="im"><br>
> reference to `mkl_serv_getversionstring_c'<br>
> /opt/intel/mkl/<a href="http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so" target="_blank">10.2.1.017/lib/em64t/libmkl_intel_lp64.so</a><br></div>
<<a href="http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so" target="_blank">http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so</a>>: undefined<div class="im"><br>
> reference to `mkl_spblas_lp64_mkl_dcsrsky'<br>
> /opt/intel/mkl/<a href="http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so" target="_blank">10.2.1.017/lib/em64t/libmkl_intel_lp64.so</a><br></div>
<<a href="http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so" target="_blank">http://10.2.1.017/lib/em64t/libmkl_intel_lp64.so</a>>: undefined<div class="im"><br>
> reference to `mkl_pdett__d_init_trig_transform'<br>
> make[1]: *** [lapw2c_mpi] Error 1<br>
> make[1]: Leaving directory `/opt/wien2k/SRC_lapw2'<br>
> make: *** [cp] Error 2<br>
><br>
><br>
> i can find no mention of the fftw3d errors anywhere.<br>
><br>
> my complete appropriate section of siteconfig<br>
><br>
> # Linux PC system with IFC 10 compiler + mkl 10 (-ip is<br>
broken; -static does<br>
> not give traceback-lines)<br>
> linuxif9:FC:ifort<br>
> linuxif9:MPF:ifort<br>
> linuxif9:CC:cc<br>
> linuxif9:FOPT:-FR -mp1 -w -prec_div -pc80 -pad -align<br>
-DINTEL_VML<br>
> -traceback -O3 -xW<br>
> linuxif9:FPOPT:$(FOPT) -FR -mp1 -w -prec_div -pc80 -pad<br>
-align -DINTEL_VML<br>
> -traceback -I/opt/mpich2/include<br>
> linuxif9:LDFLAGS: $(FOPT)<br>
-L/opt/intel/mkl/<a href="http://10.2.1.017/lib/em64t" target="_blank">10.2.1.017/lib/em64t</a><br></div>
<<a href="http://10.2.1.017/lib/em64t" target="_blank">http://10.2.1.017/lib/em64t</a>> -pthread<div class="im"><br>
> -i-static<br>
> linuxif9:R_LIBS:-L/opt/intel/mkl/<a href="http://10.2.1.017/lib/em64t" target="_blank">10.2.1.017/lib/em64t</a><br></div>
<<a href="http://10.2.1.017/lib/em64t" target="_blank">http://10.2.1.017/lib/em64t</a>> -lmkl_lapack -lmkl<div class="im"><br>
> -liomp5 -lguide -lmkl_core<br>
> linuxif9:DPARALLEL:'-DParallel'<br>
> linuxif9:RP_LIBS: -L/opt/intel/mkl/<a href="http://10.2.1.017/lib/em64t" target="_blank">10.2.1.017/lib/em64t</a><br></div>
<<a href="http://10.2.1.017/lib/em64t" target="_blank">http://10.2.1.017/lib/em64t</a>> -lmkl_lapack<div class="im"><br>
> -lmkl_intel_lp64 -lmkl_scalapack_lp64 -lmkl_blacs_lp64<br>
-lmkl_sequential<br>
> -L/opt/fftw-2.1.5/lib/ -lfftw_mpi -lfftw<br>
> linuxif9:MPIRUN:mpiexec _EXEC_<br>
><br>
><br>
> any assistance in this matter would be appreciated.<br>
><br>
> Thank you,<br>
> JD<br>
><br>
> _______________________________________________<br>
> Wien mailing list<br>
> <a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a><br></div>
<mailto:<a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a>><div class="im"><br>
> <a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien" target="_blank">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a><br>
><br>
><br>
<br>
<br>
<br>
--<br>
Laurence Marks<br>
Department of Materials Science and Engineering<br>
MSE Rm 2036 Cook Hall<br>
2220 N Campus Drive<br>
Northwestern University<br>
Evanston, IL 60208, USA<br>
Tel: (847) 491-3996 Fax: (847) 491-7820<br>
email: L-marks at northwestern dot edu<br></div>
Web: <a href="http://www.numis.northwestern.edu" target="_blank">www.numis.northwestern.edu</a> <<a href="http://www.numis.northwestern.edu" target="_blank">http://www.numis.northwestern.edu</a>><div class="im">
<br>
Chair, Commission on Electron Crystallography of IUCR<br></div>
<a href="http://www.numis.northwestern.edu/" target="_blank">www.numis.northwestern.edu/</a> <<a href="http://www.numis.northwestern.edu/" target="_blank">http://www.numis.northwestern.edu/</a>><div class="im">
<br>
Electron crystallography is the branch of science that uses electron<br>
scattering and imaging to study the structure of matter.<br>
_______________________________________________<br>
Wien mailing list<br>
<a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a><br></div>
<mailto:<a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a>><div class="im"><br>
<a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien" target="_blank">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a><br>
<br>
<br>
<br>
<br></div>
------------------------------------------------------------------------<div class="im"><br>
<br>
_______________________________________________<br>
Wien mailing list<br>
<a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a><br>
<a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien" target="_blank">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a><br>
</div></blockquote><font color="#888888">
<br>
-- <br>
<br>
P.Blaha<br>
--------------------------------------------------------------------------<br>
Peter BLAHA, Inst.f. Materials Chemistry, TU Vienna, A-1060 Vienna<br>
Phone: +43-1-58801-15671 FAX: +43-1-58801-15698<br>
Email: <a href="mailto:blaha@theochem.tuwien.ac.at" target="_blank">blaha@theochem.tuwien.ac.at</a> WWW: <a href="http://info.tuwien.ac.at/theochem/" target="_blank">http://info.tuwien.ac.at/theochem/</a><br>
--------------------------------------------------------------------------</font><div><div></div><div class="h5"><br>
_______________________________________________<br>
Wien mailing list<br>
<a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a><br>
<a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien" target="_blank">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a><br>
</div></div></blockquote></div><br>