<div dir="ltr">Thanks a lot. <div>On cluster A, RKM was automatically reduced to 4.88 while on cluster B RKM was kept at 7. I didn't expect this, though I was aware that WIEN2k would automatically reduce RKM in some cases. But is it reasonable for an iteration to run for eight hours with the following parameters?</div>
<div>Minimum sphere size: 1.65000 Bohr.</div><div>Total k-mesh : 8</div><div>Gmax : 12</div><div><br></div><div><div>:RKM : MATRIX SIZE23486LOs:1944 RKM= 7.00 WEIGHT= 2.00 PGR:</div><div>:RKM : MATRIX SIZE23486LOs:1944 RKM= 7.00 WEIGHT= 2.00 PGR:</div>
</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Oct 17, 2013 at 8:54 AM, Peter Blaha <span dir="ltr"><<a href="mailto:pblaha@theochem.tuwien.ac.at" target="_blank">pblaha@theochem.tuwien.ac.at</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">The Xeon X5550 processor is a 4 core processor and your cluster may have combined a few of them on one node (2-4 ?) Anyway, 14 cores are not really possible ??<br>
<br>
Have you done more than just looking on the total time ?<br>
<br>
Is the machines file the same on both clusters ? Such a machines file does NOT use mpi.<br>
<br>
One guess in case you really use mpi on cluster B (with a different .machines file): In the sequential run (A) the basis set is limited by NMATMAX, in the mpi-parallel run it is not (or it is scaled up by sqrt(N-core)).<br>
So it could be that run A has a MUCH smaller RKMAX than run (B).<br>
Check grep :RKM case.scf of the two runs.<br>
What are the real matrix sizes ????<br>
<br>
Alternatively, NMATMAX could be chosen differently on the two machines since somebody else installed WIEN2k.<br>
<br>
Please compare carefully the resulting case.output1_1 files and eventually send the RELEVANT PARTS OF THEM.<br>
<br>
<br>
In any case, a 72 atom cell should NOT take 2 h / iteration (or even 8 ??).<br>
<br>
What are your sphere sizes ???, what gives :RKM in case.scf ???<br>
<br>
At least one can set OMP_NUM_THREAD=2 or 4 and speed up the code by a factor of almost 2. (You should see in the dayfile something close to 200 % instead of ~100%<div class="im"><br>
> c1208-ib(1) 26558.265u 17.956s 7:34:14.39 97.5%0+0k 0+0io 0pf+0w<br>
<br></div>
In essence: A matrix size of 10000 (real, with inversion) lapw1 should take in the order of 10 min (no mpi, maybe with OMP_NUM_THREAD=2)<div><div class="h5"><br>
<br>
<br>
On 10/17/2013 04:33 PM, Yundi Quan wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
Thanks for your reply.<br>
a). both machines are set up in a way that once a node is assigned to a<br>
job, it cannot be assigned to another.<br>
b). The .machines file looks like this<br>
1:node1<br>
1:node2<br>
1:node3<br>
1:node4<br>
1:node5<br>
1:node6<br>
1:node7<br>
1:node8<br>
granularity:1<br>
extrafine:1<br>
lapw2_vector_split:1<br>
<br>
I've been trying to avoid using mpi because sometime mpi can slow down<br>
my calculations because of poor communications between nodes.<br>
<br>
c). the amount of memory available to a core does not seem to be the<br>
problem in my case because my job could run smoothly on cluster A where<br>
each node has 8G memory and 8 core). But my job runs into memory<br>
problems on cluster B where each core has much more memory available. I<br>
wonder whether there are parameters which I should change in WIEN2k to<br>
reduce the memory usage.<br>
<br>
d). My dayfile for a single iteration looks like this. The wallclocks<br>
are around 500.<br>
<br>
<br>
cycle 1 (Fri Oct 11 02:14:05 PDT 2013) (40/99 to go)<br>
<br></div></div>
> lapw0 -p(02:14:05) starting parallel lapw0 at Fri Oct 11 02:14:06<div class="im"><br>
PDT 2013<br>
-------- .machine0 : processors<br>
running lapw0 in single mode<br></div>
1431.414u 22.267s 24:14.84 99.9%0+0k 0+0io 0pf+0w<br>
> lapw1 -up -p -c(02:38:20) starting parallel lapw1 at Fri Oct 11<div class="im"><br>
02:38:20 PDT 2013<br>
-> starting parallel LAPW1 jobs at Fri Oct 11 02:38:21 PDT 2013<br>
running LAPW1 in parallel mode (using .machines)<br>
8 number_of_parallel_jobs<br></div>
c1208-ib(1) 26558.265u 17.956s 7:34:14.39 97.5%0+0k 0+0io 0pf+0w<br>
c1201-ib(1) 26845.212u 15.496s 7:39:59.37 97.3%0+0k 0+0io 0pf+0w<br>
c1180-ib(1) 25872.609u 18.143s 7:23:53.43 97.2%0+0k 0+0io 0pf+0w<br>
c1179-ib(1) 26040.482u 17.868s 7:26:38.66 97.2%0+0k 0+0io 0pf+0w<br>
c1178-ib(1) 26571.271u 17.946s 7:34:16.23 97.5%0+0k 0+0io 0pf+0w<br>
c1177-ib(1) 27108.070u 34.294s 8:32:55.53 88.1%0+0k 0+0io 0pf+0w<br>
c1171-ib(1) 26729.399u 14.175s 7:36:22.67 97.6%0+0k 0+0io 0pf+0w<br>
c0844-ib(1) 25883.863u 47.148s 8:12:35.54 87.7%0+0k 0+0io 0pf+0w<div class="im"><br>
Summary of lapw1para:<br>
c1208-ibk=1user=26558.<u></u>3wallclock=454<br>
c1201-ibk=1user=26845.<u></u>2wallclock=459<br>
c1180-ibk=1user=25872.<u></u>6wallclock=443<br>
c1179-ibk=1user=26040.<u></u>5wallclock=446<br>
c1178-ibk=1user=26571.<u></u>3wallclock=454<br>
c1177-ibk=1user=27108.<u></u>1wallclock=512<br>
c1171-ibk=1user=26729.<u></u>4wallclock=456<br>
c0844-ibk=1user=25883.<u></u>9wallclock=492<br></div>
97.935u 34.265s 8:32:58.38 0.4%0+0k 0+0io 0pf+0w<br>
> lapw1 -dn -p -c(11:11:19) starting parallel lapw1 at Fri Oct 11<div class="im"><br>
11:11:19 PDT 2013<br>
-> starting parallel LAPW1 jobs at Fri Oct 11 11:11:19 PDT 2013<br>
running LAPW1 in parallel mode (using .machines.help)<br>
8 number_of_parallel_jobs<br></div>
c1208-ib(1) 26474.686u 16.142s 7:33:36.01 97.3%0+0k 0+0io 0pf+0w<br>
c1201-ib(1) 26099.149u 40.330s 8:04:42.58 89.8%0+0k 0+0io 0pf+0w<br>
c1180-ib(1) 26809.287u 14.724s 7:38:56.52 97.4%0+0k 0+0io 0pf+0w<br>
c1179-ib(1) 26007.527u 17.959s 7:26:10.62 97.2%0+0k 0+0io 0pf+0w<br>
c1178-ib(1) 26565.723u 17.576s 7:35:20.11 97.3%0+0k 0+0io 0pf+0w<br>
c1177-ib(1) 27114.619u 31.180s 8:21:28.34 90.2%0+0k 0+0io 0pf+0w<br>
c1171-ib(1) 26474.665u 15.309s 7:33:38.15 97.3%0+0k 0+0io 0pf+0w<br>
c0844-ib(1) 26586.569u 15.010s 7:35:22.88 97.3%0+0k 0+0io 0pf+0w<br>
Summary of lapw1para:<br>
c1208-ibk=1user=26474.<u></u>7wallclock=453<br>
c1201-ibk=1user=26099.<u></u>1wallclock=484<br>
c1180-ibk=1user=26809.<u></u>3wallclock=458<br>
c1179-ibk=1user=26007.<u></u>5wallclock=446<br>
c1178-ibk=1user=26565.<u></u>7wallclock=455<br>
c1177-ibk=1user=27114.<u></u>6wallclock=501<br>
c1171-ibk=1user=26474.<u></u>7wallclock=453<br>
c0844-ibk=1user=26586.<u></u>6wallclock=455<br>
104.607u 18.798s 8:21:30.92 0.4%0+0k 0+0io 0pf+0w<div class="im"><br>
> lapw2 -up -p -c (19:32:50) running LAPW2 in parallel mode<br>
c1208-ib 1016.517u 13.674s 17:11.10 99.9% 0+0k 0+0io 0pf+0w<br>
c1201-ib 1017.359u 13.669s 17:11.82 99.9% 0+0k 0+0io 0pf+0w<br>
c1180-ib 1033.056u 13.283s 17:27.07 99.9% 0+0k 0+0io 0pf+0w<br>
c1179-ib 1037.551u 13.447s 17:31.50 99.9% 0+0k 0+0io 0pf+0w<br>
c1178-ib 1019.156u 13.729s 17:13.49 99.9% 0+0k 0+0io 0pf+0w<br>
c1177-ib 1021.878u 13.731s 17:16.07 99.9% 0+0k 0+0io 0pf+0w<br>
c1171-ib 1032.417u 13.681s 17:26.70 99.9% 0+0k 0+0io 0pf+0w<br>
c0844-ib 1022.315u 13.870s 17:16.81 99.9% 0+0k 0+0io 0pf+0w<br>
Summary of lapw2para:<br>
c1208-ibuser=1016.52wallclock=<u></u>1031.1<br>
c1201-ibuser=1017.36wallclock=<u></u>1031.82<br>
c1180-ibuser=1033.06wallclock=<u></u>1047.07<br>
c1179-ibuser=1037.55wallclock=<u></u>1051.5<br>
c1178-ibuser=1019.16wallclock=<u></u>1033.49<br>
c1177-ibuser=1021.88wallclock=<u></u>1036.07<br>
c1171-ibuser=1032.42wallclock=<u></u>1046.7<br>
c0844-ibuser=1022.32wallclock=<u></u>1036.81<br></div>
31.923u 13.526s 18:20.12 4.1%0+0k 0+0io 0pf+0w<div class="im"><br>
> lapw2 -dn -p -c (19:51:10) running LAPW2 in parallel mode<br>
c1208-ib 947.942u 13.364s 16:01.75 99.9% 0+0k 0+0io 0pf+0w<br>
c1201-ib 932.766u 13.640s 15:49.22 99.7% 0+0k 0+0io 0pf+0w<br>
c1180-ib 932.474u 13.609s 15:47.76 99.8% 0+0k 0+0io 0pf+0w<br>
c1179-ib 936.171u 13.691s 15:50.33 99.9% 0+0k 0+0io 0pf+0w<br>
c1178-ib 947.798u 13.493s 16:04.99 99.6% 0+0k 0+0io 0pf+0w<br>
c1177-ib 947.786u 13.350s 16:04.89 99.6% 0+0k 0+0io 0pf+0w<br>
c1171-ib 930.971u 13.874s 15:45.22 99.9% 0+0k 0+0io 0pf+0w<br>
c0844-ib 950.723u 13.426s 16:04.69 99.9% 0+0k 0+0io 0pf+0w<br>
Summary of lapw2para:<br>
c1208-ibuser=947.942wallclock=<u></u>961.75<br>
c1201-ibuser=932.766wallclock=<u></u>949.22<br>
c1180-ibuser=932.474wallclock=<u></u>947.76<br>
c1179-ibuser=936.171wallclock=<u></u>950.33<br>
c1178-ibuser=947.798wallclock=<u></u>964.99<br>
c1177-ibuser=947.786wallclock=<u></u>964.89<br>
c1171-ibuser=930.971wallclock=<u></u>945.22<br>
c0844-ibuser=950.723wallclock=<u></u>964.69<br>
31.522u 13.879s 16:53.13 4.4%0+0k 0+0io 0pf+0w<br></div>
> lcore -up(20:08:03) 2.993u 0.587s 0:03.75 95.2%0+0k 0+0io 0pf+0w<br>
> lcore -dn(20:08:07) 2.843u 0.687s 0:03.66 96.1%0+0k 0+0io 0pf+0w<br>
> mixer (20:08:21) 23.206u 32.513s 0:56.63 98.3%0+0k 0+0io 0pf+0w<div class="im"><br>
:ENERGY convergence: 0 0.00001 416.9302585700000000<br>
:CHARGE convergence: 0 0.0000 3.6278086<br>
<br>
<br>
On Thu, Oct 17, 2013 at 7:11 AM, Laurence Marks<br></div><div><div class="h5">
<<a href="mailto:L-marks@northwestern.edu" target="_blank">L-marks@northwestern.edu</a> <mailto:<a href="mailto:L-marks@northwestern.edu" target="_blank">L-marks@northwestern.<u></u>edu</a>>> wrote:<br>
<br>
There are so many possibilities, a few:<br>
<br>
a) If you only request 1 core/node most queuing systems (qsub/msub<br>
etc) will allocate the other cores to other jobs. You are then going<br>
to be very dependent upon what those other jobs are doing. Normal is<br>
to use all the cores on a given node.<br>
<br>
b) When you run on cluster B, in addition to a) it is going to be<br>
inefficient to run with mpi communications across nodes and it is much<br>
better to run on a given node across cores. Are you using a machines<br>
file with eight 1: nodeA lines (for instance) or one with a single 1:<br>
nodeA nodeB....? The first does not use mpi, the second does. To use<br>
mpi within a node you would use lines such as 1:node:8. Knowledge of<br>
your .machines file will help people assist you.<br>
<br>
c) The memory on those clusters is very small, whoever bought them was<br>
not thinking about large scale jobs. I look for at least 4G/core, and<br>
2G/core is barely acceptable. You are going to have to use mpi.<br>
<br>
d) All mpi is equal, but some mpi is more equal than others. Depending<br>
upon whether you have infiniband, ethernet, openmpi, impi and how<br>
everything was compiled you can see enormous differences. One thing to<br>
look at is the difference between the cpu time and wall time (both in<br>
case.dayfile and at the bottom of case.output1_*). With a good mpi<br>
setup the wall time should be 5-10% more than the cpu time; with a bad<br>
setup it can be several times it.<br>
<br>
On Thu, Oct 17, 2013 at 8:44 AM, Yundi Quan <<a href="mailto:quanyundi@gmail.com" target="_blank">quanyundi@gmail.com</a><br></div></div><div><div class="h5">
<mailto:<a href="mailto:quanyundi@gmail.com" target="_blank">quanyundi@gmail.com</a>>> wrote:<br>
> Hi,<br>
> I have access to two clusters as a low-level user. One cluster<br>
(cluster A)<br>
> consists of nodes with 8 core and 8 G mem per node. The other cluster<br>
> (cluster B) has 24G mem per node and each node has 14 cores or<br>
more. The<br>
> cores on cluster A are Xeon CPU E5620@2.40GHz, while the cores on<br>
cluster B<br>
> are Xeon CPU X5550@2.67GH. From the specifications (2.40GHz+12288<br>
KB cache<br>
> vs 2.67GHz+8192 KB cache), two machines should be very close in<br>
performance.<br>
> But it does not seem to be so.<br>
><br>
> I have job with 72 atoms per unit cell. I initialized the job on<br>
cluster A<br>
> and ran it for a few iterations. Each iteration took 2 hours.<br>
Then, I moved<br>
> the job to cluster B (14 cores per node with @2.67GHz). Now it<br>
takes more<br>
> than 8 hours to finish one iteration. On both clusters, I request<br>
one core<br>
> per node and 8 nodes per job ( 8 is the number of k points). I<br>
compiled<br>
> WIEN2k_13 on cluster A without mpi. On cluster B, WIEN2k_12 was<br>
compiled by<br>
> the administrator with mpi.<br>
><br>
> What could have caused poor performance of cluster B? Is it<br>
because of MPI?<br>
><br>
> On an unrelated question. Sometimes memory would run out on<br>
cluster B which<br>
> has 24Gmem per node. Nevertheless the same job could run smoothly<br>
on cluster<br>
> A which only has 8 G per node.<br>
><br>
> Thanks.<br>
<br>
<br>
<br>
--<br>
Professor Laurence Marks<br>
Department of Materials Science and Engineering<br>
Northwestern University<br></div></div>
<a href="http://www.numis.northwestern.edu" target="_blank">www.numis.northwestern.edu</a> <<a href="http://www.numis.northwestern.edu" target="_blank">http://www.numis.<u></u>northwestern.edu</a>><br>
<a href="tel:1-847-491-3996" value="+18474913996" target="_blank">1-847-491-3996</a> <tel:<a href="tel:1-847-491-3996" value="+18474913996" target="_blank">1-847-491-3996</a>><div class="im"><br>
"Research is to see what everybody else has seen, and to think what<br>
nobody else has thought"<br>
Albert Szent-Gyorgi<br>
______________________________<u></u>_________________<br>
Wien mailing list<br></div>
<a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.<u></u>at</a> <mailto:<a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.<u></u>tuwien.ac.at</a>><div class="im">
<br>
<a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien" target="_blank">http://zeus.theochem.tuwien.<u></u>ac.at/mailman/listinfo/wien</a><br>
SEARCH the MAILING-LIST at:<br>
<a href="http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html" target="_blank">http://www.mail-archive.com/<u></u>wien@zeus.theochem.tuwien.ac.<u></u>at/index.html</a><br>
<br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
Wien mailing list<br>
<a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.<u></u>at</a><br>
<a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien" target="_blank">http://zeus.theochem.tuwien.<u></u>ac.at/mailman/listinfo/wien</a><br>
SEARCH the MAILING-LIST at: <a href="http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html" target="_blank">http://www.mail-archive.com/<u></u>wien@zeus.theochem.tuwien.ac.<u></u>at/index.html</a><br>
<br>
</div></blockquote><span class="HOEnZb"><font color="#888888">
<br>
-- <br>
<br>
P.Blaha<br>
------------------------------<u></u>------------------------------<u></u>--------------<br>
Peter BLAHA, Inst.f. Materials Chemistry, TU Vienna, A-1060 Vienna<br>
Phone: <a href="tel:%2B43-1-58801-165300" value="+43158801165300" target="_blank">+43-1-58801-165300</a> FAX: <a href="tel:%2B43-1-58801-165982" value="+43158801165982" target="_blank">+43-1-58801-165982</a><br>
Email: <a href="mailto:blaha@theochem.tuwien.ac.at" target="_blank">blaha@theochem.tuwien.ac.at</a> WWW: <a href="http://info.tuwien.ac.at/theochem/" target="_blank">http://info.tuwien.ac.at/<u></u>theochem/</a><br>
------------------------------<u></u>------------------------------<u></u>--------------</font></span><div class="HOEnZb"><div class="h5"><br>
______________________________<u></u>_________________<br>
Wien mailing list<br>
<a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.<u></u>at</a><br>
<a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien" target="_blank">http://zeus.theochem.tuwien.<u></u>ac.at/mailman/listinfo/wien</a><br>
SEARCH the MAILING-LIST at: <a href="http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html" target="_blank">http://www.mail-archive.com/<u></u>wien@zeus.theochem.tuwien.ac.<u></u>at/index.html</a><br>
</div></div></blockquote></div><br></div>