<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Thank you very much for your suggestion. I actually managed to figure this out by myself an hour or so ago. At the same time (usually not a good idea) I also compiled the mkl interface for fftw2 rather than using the standalone version I had compiled by myself earlier. Thus the RP library line looks like<div><br></div><div>-lmkl_scalapack_lp64 -lmkl_solver_lp64 -lmkl_blacs_intelmpi_lp64 /opt/intel/mkl/lib/intel64/libfftw2x_cdft_DOUBLE.a $(R_LIBS)</div><div><br></div><div>All compiles <br><div><br></div><div>In the past ten minutes I have been running the TiC example under mpi and I must say the mpi version seems tricker to use than I first though as each mpi process seems to use multiple threads and it is easy to over saturate the number of cores one has. I have been testing the mpi version on a single node (32 cores) of a 500 core cluster to get a feel for things.</div><div><br></div><div>Running the TiC calculation (with 4000 k-points) in serial mode (the mkl routine uses multiple threads) took about 75 seconds</div><div><br></div><div>Running the sample calculation in parallel mode (with 5 mpi processes each which spawn multiple threads for an average cpu load of 28 in a 32 core machine) takes longer, specifically 300 seconds.</div><div><br></div><div>Carrying out the same calculation with "1:localhost* on six separate lines (e.g. lapw runs six processes locally and then aggregates without using mpi) finishes quickest of all in about 60 seconds.</div><div><br></div><div><br></div><div>I don't really understand what is going on. Obviously I don't really need to use the mpi/or parallel calculation for small problems as in the TiC example, but I would like to use it for larger supercells. Is there some sort of guiding principle which will help me decide which approach is best? I assume the parallelization by running multiple jobs is also limited to a single machine so this won't scale very well.</div><div><br></div><div>Thanks for any advice.</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Paul</div><div><br></div><div>On Jan 23, 2012, at 8:08 AM, Laurence Marks wrote:</div><div><div><br class="Apple-interchange-newline"><blockquote type="cite"><div>A guess: you are using the wrong version of blacs. You need a<br>-lmkl_blacs_intelmpi_XX<br>where "XX" is the one for your system. I have seen this give the same error.<br><br>Use <a href="http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/">http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/</a><br><br>For reference, with openmpi it is _openmpi_ instead of _intelmpi_, and<br>similarly for sgi.<br><br>2012/1/22 Paul Fons <<a href="mailto:paul-fons@aist.go.jp">paul-fons@aist.go.jp</a>>:<br><blockquote type="cite"><br></blockquote><blockquote type="cite">Hi,<br></blockquote><blockquote type="cite">I have Wien2K running on a cluster of linux boxes each with 32 cores and<br></blockquote><blockquote type="cite">connected by 10Gb ethernet. I have compiled Wien2K by the 3.174 version of<br></blockquote><blockquote type="cite">Wien2K (I learned the hard way that bugs in the newer versions of the Intel<br></blockquote><blockquote type="cite">compiler lead to crashes in Wien2K). I have also installed Intel's MPI.<br></blockquote><blockquote type="cite"> First, the single process Wien2K, let's say for the TiC case, works fine.<br></blockquote><blockquote type="cite"> It also works fine when I use a .machines file like<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">granulaity:1<br></blockquote><blockquote type="cite">localhost:1<br></blockquote><blockquote type="cite">localhost:1<br></blockquote><blockquote type="cite">… (24 times).<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">This file leads to parallel execution without error. I can vary the number<br></blockquote><blockquote type="cite">of processes by increasing the number of localhost:1 and the number of<br></blockquote><blockquote type="cite">localhost:1 lines in the file and still everything works fine. When I try<br></blockquote><blockquote type="cite">to use mpi to communicate with one process, it works as well.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">1:localhost:1<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">lstarting parallel lapw1 at Mon Jan 23 06:49:16 JST 2012<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">-> starting parallel LAPW1 jobs at Mon Jan 23 06:49:16 JST 2012<br></blockquote><blockquote type="cite">running LAPW1 in parallel mode (using .machines)<br></blockquote><blockquote type="cite">1 number_of_parallel_jobs<br></blockquote><blockquote type="cite">[1] 22417<br></blockquote><blockquote type="cite"> LAPW1 END<br></blockquote><blockquote type="cite">[1] + Done ( cd $PWD; $t $exe ${def}_$loop.def; rm<br></blockquote><blockquote type="cite">-f .lock_$lockfile[$p] ) >> .time1_$loop<br></blockquote><blockquote type="cite"> localhost(111) 179.004u 4.635s 0:32.73 561.0%<span class="Apple-tab-span" style="white-space:pre"> </span>0+0k 0+26392io 0pf+0w<br></blockquote><blockquote type="cite"> Summary of lapw1para:<br></blockquote><blockquote type="cite"> localhost<span class="Apple-tab-span" style="white-space:pre"> </span> k=111<span class="Apple-tab-span" style="white-space:pre"> </span> user=179.004<span class="Apple-tab-span" style="white-space:pre"> </span> wallclock=32.73<br></blockquote><blockquote type="cite">179.167u 4.791s 0:35.61 516.5%<span class="Apple-tab-span" style="white-space:pre"> </span>0+0k 0+26624io 0pf+0w<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Changing the machine file to use more than one process (the same form of<br></blockquote><blockquote type="cite">error occurs for more than 2)<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">1:localhost:2<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">lead to a run time error in the MPI subsystem.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">starting parallel lapw1 at Mon Jan 23 06:51:04 JST 2012<br></blockquote><blockquote type="cite">-> starting parallel LAPW1 jobs at Mon Jan 23 06:51:04 JST 2012<br></blockquote><blockquote type="cite">running LAPW1 in parallel mode (using .machines)<br></blockquote><blockquote type="cite">1 number_of_parallel_jobs<br></blockquote><blockquote type="cite">[1] 22673<br></blockquote><blockquote type="cite">Fatal error in MPI_Comm_size: Invalid communicator, error stack:<br></blockquote><blockquote type="cite">MPI_Comm_size(123): MPI_Comm_size(comm=0x5b, size=0x7ed20c) failed<br></blockquote><blockquote type="cite">MPI_Comm_size(76).: Invalid communicator<br></blockquote><blockquote type="cite">Fatal error in MPI_Comm_size: Invalid communicator, error stack:<br></blockquote><blockquote type="cite">MPI_Comm_size(123): MPI_Comm_size(comm=0x5b, size=0x7ed20c) failed<br></blockquote><blockquote type="cite">MPI_Comm_size(76).: Invalid communicator<br></blockquote><blockquote type="cite">[1] + Done ( cd $PWD; $t $ttt; rm -f<br></blockquote><blockquote type="cite">.lock_$lockfile[$p] ) >> .time1_$loop<br></blockquote><blockquote type="cite"> localhost localhost(111) APPLICATION TERMINATED WITH THE EXIT STRING:<br></blockquote><blockquote type="cite">Hangup (signal 1)<br></blockquote><blockquote type="cite">0.037u 0.036s 0:00.06 100.0%<span class="Apple-tab-span" style="white-space:pre"> </span>0+0k 0+0io 0pf+0w<br></blockquote><blockquote type="cite">TiC.scf1_1: No such file or directory.<br></blockquote><blockquote type="cite"> Summary of lapw1para:<br></blockquote><blockquote type="cite"> localhost<span class="Apple-tab-span" style="white-space:pre"> </span> k=0<span class="Apple-tab-span" style="white-space:pre"> </span> user=111<span class="Apple-tab-span" style="white-space:pre"> </span> wallclock=0<br></blockquote><blockquote type="cite">0.105u 0.168s 0:03.21 8.0%<span class="Apple-tab-span" style="white-space:pre"> </span>0+0k 0+216io 0pf+0w<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">I have properly sourced the appropriate runtime environment for the Intel<br></blockquote><blockquote type="cite">system. For example, compiling (mpiifort) and running the f90 mpi test<br></blockquote><blockquote type="cite">program from intel produces:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">mpirun -np 32 /home/paulfons/mpitest/testf90<br></blockquote><blockquote type="cite"> Hello world: rank 0 of 32 running on<br></blockquote><blockquote type="cite"> asccmp177<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> Hello world: rank 1 of 32 running on (32 times)<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Does anyone have any suggestions as to what to try next? I am not sure how<br></blockquote><blockquote type="cite">to debug things from here. I have about 512 nodes that I can use for larger<br></blockquote><blockquote type="cite">calculations that only can be accessed by mpi (the ssh setup works fine as<br></blockquote><blockquote type="cite">well by the way). It would be great to figure out what is wrong.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Thanks.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Dr. Paul Fons<br></blockquote><blockquote type="cite">Functional Nano-phase-change Research Team<br></blockquote><blockquote type="cite">Team Leader<br></blockquote><blockquote type="cite">Nanodevice Innovation Research Center (NIRC)<br></blockquote><blockquote type="cite">National Institute for Advanced Industrial Science & Technology<br></blockquote><blockquote type="cite">METI<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">AIST Central 4, Higashi 1-1-1<br></blockquote><blockquote type="cite">Tsukuba, Ibaraki JAPAN 305-8568<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">tel. +81-298-61-5636<br></blockquote><blockquote type="cite">fax. +81-298-61-2939<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">email: <a href="mailto:paul-fons@aist.go.jp">paul-fons@aist.go.jp</a><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">The following lines are in a Japanese font<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">〒305-8562 茨城県つくば市つくば中央東 1-1-1<br></blockquote><blockquote type="cite">産業技術総合研究所<br></blockquote><blockquote type="cite">ナノ電子デバイス研究センター<br></blockquote><blockquote type="cite">相変化新規機能デバイス研究チーム チームリーダー<br></blockquote><blockquote type="cite">ポール・フォンス<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">_______________________________________________<br></blockquote><blockquote type="cite">Wien mailing list<br></blockquote><blockquote type="cite"><a href="mailto:Wien@zeus.theochem.tuwien.ac.at">Wien@zeus.theochem.tuwien.ac.at</a><br></blockquote><blockquote type="cite"><a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a><br></blockquote><blockquote type="cite"><br></blockquote><br><br><br>-- <br>Professor Laurence Marks<br>Department of Materials Science and Engineering<br>Northwestern University<br><a href="http://www.numis.northwestern.edu">www.numis.northwestern.edu</a> 1-847-491-3996<br>"Research is to see what everybody else has seen, and to think what<br>nobody else has thought"<br>Albert Szent-Gyorgi<br>_______________________________________________<br>Wien mailing list<br><a href="mailto:Wien@zeus.theochem.tuwien.ac.at">Wien@zeus.theochem.tuwien.ac.at</a><br>http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien<br></div></blockquote></div><br><div apple-content-edited="true">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; "><font face="Helvetica" size="3" style="font: normal normal normal 12px/normal Helvetica; ">Dr. Paul Fons</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; ">Functional Nano-phase-change Research Team</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; ">Team Leader</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; ">Nanodevice Innovation Research Center (NIRC)</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; "><font face="Helvetica" size="3" style="font: normal normal normal 12px/normal Helvetica; ">National Institute for Advanced Industrial Science & Technology</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; "><font face="Helvetica" size="3" style="font: normal normal normal 12px/normal Helvetica; ">METI</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; "><font face="Helvetica" size="3" style="font: normal normal normal 12px/normal Helvetica; ">AIST Central 4, Higashi 1-1-1</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; "><font face="Helvetica" size="3" style="font: normal normal normal 12px/normal Helvetica; ">Tsukuba, Ibaraki JAPAN 305-8568</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; min-height: 14px; font-size: 12px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; "><font face="Helvetica" size="3" style="font: normal normal normal 12px/normal Helvetica; ">tel. +81-298-61-5636</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; "><font face="Helvetica" size="3" style="font: normal normal normal 12px/normal Helvetica; ">fax. +81-298-61-2939</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; min-height: 14px; font-size: 12px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; "><font face="Helvetica" size="3" style="font: normal normal normal 12px/normal Helvetica; ">email: </font><font face="Helvetica" size="3" color="#1919ff" style="font: normal normal normal 12px/normal Helvetica; color: rgb(25, 25, 255); "><u style="color: rgb(25, 25, 255); -webkit-text-decorations-in-effect: underline; "><a href="mailto:paul-fons@aist.go.jp">paul-fons@aist.go.jp</a></u></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; min-height: 14px; font-size: 12px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; "><font class="Apple-style-span" face="Hiragino Kaku Gothic Pro">The following lines are in a Japanese font</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; "><font class="Apple-style-span" face="Hiragino Kaku Gothic Pro"><br class="khtml-block-placeholder"></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; "><font class="Apple-style-span" face="Hiragino Kaku Gothic Pro">〒305-8562 茨城県つくば市つくば中央東 1-1-1</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; "><font class="Apple-style-span" face="Hiragino Kaku Gothic Pro">産業技術総合研究所</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; "><font class="Apple-style-span" face="Hiragino Kaku Gothic Pro"><span class="Apple-style-span" style="font-family: 'Hiragino Kaku Gothic Pro'; ">ナノ電子デバイス研究センター</span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; "><font class="Apple-style-span" face="'Hiragino Kaku Gothic Pro'">相変化新規機能デバイス研究チーム チームリーダー</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; "><span class="Apple-style-span" style="font-family: 'Hiragino Kaku Gothic Pro'; ">ポール・フォンス</span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; "><br class="khtml-block-placeholder"></div></div></div><br class="Apple-interchange-newline"><br class="Apple-interchange-newline">
</div>
<br></div></div></body></html>