<div dir="ltr">Thanks for your reply.<div>a). both machines are set up in a way that once a node is assigned to a job, it cannot be assigned to another. </div><div>b). The .machines file looks like this</div><div><div>1:node1</div>
<div>1:node2</div><div>1:node3</div><div>1:node4</div><div>1:node5</div><div>1:node6</div><div>1:node7</div><div>1:node8</div><div>granularity:1</div><div>extrafine:1</div><div>lapw2_vector_split:1</div></div><div><br></div>
<div>I've been trying to avoid using mpi because sometime mpi can slow down my calculations because of poor communications between nodes. </div><div><br></div><div>c). the amount of memory available to a core does not seem to be the problem in my case because my job could run smoothly on cluster A where each node has 8G memory and 8 core). But my job runs into memory problems on cluster B where each core has much more memory available. I wonder whether there are parameters which I should change in WIEN2k to reduce the memory usage. </div>
<div><br></div><div>d). My dayfile for a single iteration looks like this. The wallclocks are around 500.</div><div><br></div><div><br></div><div><div> cycle 1 <span class="" style="white-space:pre"> </span>(Fri Oct 11 02:14:05 PDT 2013) <span class="" style="white-space:pre"> </span>(40/99 to go)</div>
<div><br></div><div>> lapw0 -p<span class="" style="white-space:pre"> </span>(02:14:05) starting parallel lapw0 at Fri Oct 11 02:14:06 PDT 2013</div><div>-------- .machine0 : processors</div><div>running lapw0 in single mode</div>
<div>1431.414u 22.267s 24:14.84 99.9%<span class="" style="white-space:pre"> </span>0+0k 0+0io 0pf+0w</div><div>> lapw1 -up -p -c<span class="" style="white-space:pre"> </span>(02:38:20) starting parallel lapw1 at Fri Oct 11 02:38:20 PDT 2013</div>
<div>-> starting parallel LAPW1 jobs at Fri Oct 11 02:38:21 PDT 2013</div><div>running LAPW1 in parallel mode (using .machines)</div><div>8 number_of_parallel_jobs</div><div> c1208-ib(1) 26558.265u 17.956s 7:34:14.39 97.5%<span class="" style="white-space:pre"> </span>0+0k 0+0io 0pf+0w</div>
<div> c1201-ib(1) 26845.212u 15.496s 7:39:59.37 97.3%<span class="" style="white-space:pre"> </span>0+0k 0+0io 0pf+0w</div><div> c1180-ib(1) 25872.609u 18.143s 7:23:53.43 97.2%<span class="" style="white-space:pre"> </span>0+0k 0+0io 0pf+0w</div>
<div> c1179-ib(1) 26040.482u 17.868s 7:26:38.66 97.2%<span class="" style="white-space:pre"> </span>0+0k 0+0io 0pf+0w</div><div> c1178-ib(1) 26571.271u 17.946s 7:34:16.23 97.5%<span class="" style="white-space:pre"> </span>0+0k 0+0io 0pf+0w</div>
<div> c1177-ib(1) 27108.070u 34.294s 8:32:55.53 88.1%<span class="" style="white-space:pre"> </span>0+0k 0+0io 0pf+0w</div><div> c1171-ib(1) 26729.399u 14.175s 7:36:22.67 97.6%<span class="" style="white-space:pre"> </span>0+0k 0+0io 0pf+0w</div>
<div> c0844-ib(1) 25883.863u 47.148s 8:12:35.54 87.7%<span class="" style="white-space:pre"> </span>0+0k 0+0io 0pf+0w</div><div> Summary of lapw1para:</div><div> c1208-ib<span class="" style="white-space:pre"> </span> k=1<span class="" style="white-space:pre"> </span> user=26558.3<span class="" style="white-space:pre"> </span> wallclock=454</div>
<div> c1201-ib<span class="" style="white-space:pre"> </span> k=1<span class="" style="white-space:pre"> </span> user=26845.2<span class="" style="white-space:pre"> </span> wallclock=459</div><div> c1180-ib<span class="" style="white-space:pre"> </span> k=1<span class="" style="white-space:pre"> </span> user=25872.6<span class="" style="white-space:pre"> </span> wallclock=443</div>
<div> c1179-ib<span class="" style="white-space:pre"> </span> k=1<span class="" style="white-space:pre"> </span> user=26040.5<span class="" style="white-space:pre"> </span> wallclock=446</div><div> c1178-ib<span class="" style="white-space:pre"> </span> k=1<span class="" style="white-space:pre"> </span> user=26571.3<span class="" style="white-space:pre"> </span> wallclock=454</div>
<div> c1177-ib<span class="" style="white-space:pre"> </span> k=1<span class="" style="white-space:pre"> </span> user=27108.1<span class="" style="white-space:pre"> </span> wallclock=512</div><div> c1171-ib<span class="" style="white-space:pre"> </span> k=1<span class="" style="white-space:pre"> </span> user=26729.4<span class="" style="white-space:pre"> </span> wallclock=456</div>
<div> c0844-ib<span class="" style="white-space:pre"> </span> k=1<span class="" style="white-space:pre"> </span> user=25883.9<span class="" style="white-space:pre"> </span> wallclock=492</div><div>97.935u 34.265s 8:32:58.38 0.4%<span class="" style="white-space:pre"> </span>0+0k 0+0io 0pf+0w</div>
<div>> lapw1 -dn -p -c<span class="" style="white-space:pre"> </span>(11:11:19) starting parallel lapw1 at Fri Oct 11 11:11:19 PDT 2013</div><div>-> starting parallel LAPW1 jobs at Fri Oct 11 11:11:19 PDT 2013</div>
<div>running LAPW1 in parallel mode (using .machines.help)</div><div>8 number_of_parallel_jobs</div><div> c1208-ib(1) 26474.686u 16.142s 7:33:36.01 97.3%<span class="" style="white-space:pre"> </span>0+0k 0+0io 0pf+0w</div>
<div> c1201-ib(1) 26099.149u 40.330s 8:04:42.58 89.8%<span class="" style="white-space:pre"> </span>0+0k 0+0io 0pf+0w</div><div> c1180-ib(1) 26809.287u 14.724s 7:38:56.52 97.4%<span class="" style="white-space:pre"> </span>0+0k 0+0io 0pf+0w</div>
<div> c1179-ib(1) 26007.527u 17.959s 7:26:10.62 97.2%<span class="" style="white-space:pre"> </span>0+0k 0+0io 0pf+0w</div><div> c1178-ib(1) 26565.723u 17.576s 7:35:20.11 97.3%<span class="" style="white-space:pre"> </span>0+0k 0+0io 0pf+0w</div>
<div> c1177-ib(1) 27114.619u 31.180s 8:21:28.34 90.2%<span class="" style="white-space:pre"> </span>0+0k 0+0io 0pf+0w</div><div> c1171-ib(1) 26474.665u 15.309s 7:33:38.15 97.3%<span class="" style="white-space:pre"> </span>0+0k 0+0io 0pf+0w</div>
<div> c0844-ib(1) 26586.569u 15.010s 7:35:22.88 97.3%<span class="" style="white-space:pre"> </span>0+0k 0+0io 0pf+0w</div><div> Summary of lapw1para:</div><div> c1208-ib<span class="" style="white-space:pre"> </span> k=1<span class="" style="white-space:pre"> </span> user=26474.7<span class="" style="white-space:pre"> </span> wallclock=453</div>
<div> c1201-ib<span class="" style="white-space:pre"> </span> k=1<span class="" style="white-space:pre"> </span> user=26099.1<span class="" style="white-space:pre"> </span> wallclock=484</div><div> c1180-ib<span class="" style="white-space:pre"> </span> k=1<span class="" style="white-space:pre"> </span> user=26809.3<span class="" style="white-space:pre"> </span> wallclock=458</div>
<div> c1179-ib<span class="" style="white-space:pre"> </span> k=1<span class="" style="white-space:pre"> </span> user=26007.5<span class="" style="white-space:pre"> </span> wallclock=446</div><div> c1178-ib<span class="" style="white-space:pre"> </span> k=1<span class="" style="white-space:pre"> </span> user=26565.7<span class="" style="white-space:pre"> </span> wallclock=455</div>
<div> c1177-ib<span class="" style="white-space:pre"> </span> k=1<span class="" style="white-space:pre"> </span> user=27114.6<span class="" style="white-space:pre"> </span> wallclock=501</div><div> c1171-ib<span class="" style="white-space:pre"> </span> k=1<span class="" style="white-space:pre"> </span> user=26474.7<span class="" style="white-space:pre"> </span> wallclock=453</div>
<div> c0844-ib<span class="" style="white-space:pre"> </span> k=1<span class="" style="white-space:pre"> </span> user=26586.6<span class="" style="white-space:pre"> </span> wallclock=455</div><div>104.607u 18.798s 8:21:30.92 0.4%<span class="" style="white-space:pre"> </span>0+0k 0+0io 0pf+0w</div>
<div>> lapw2 -up -p -c <span class="" style="white-space:pre"> </span>(19:32:50) running LAPW2 in parallel mode</div><div> c1208-ib 1016.517u 13.674s 17:11.10 99.9% 0+0k 0+0io 0pf+0w</div><div> c1201-ib 1017.359u 13.669s 17:11.82 99.9% 0+0k 0+0io 0pf+0w</div>
<div> c1180-ib 1033.056u 13.283s 17:27.07 99.9% 0+0k 0+0io 0pf+0w</div><div> c1179-ib 1037.551u 13.447s 17:31.50 99.9% 0+0k 0+0io 0pf+0w</div><div> c1178-ib 1019.156u 13.729s 17:13.49 99.9% 0+0k 0+0io 0pf+0w</div>
<div> c1177-ib 1021.878u 13.731s 17:16.07 99.9% 0+0k 0+0io 0pf+0w</div><div> c1171-ib 1032.417u 13.681s 17:26.70 99.9% 0+0k 0+0io 0pf+0w</div><div> c0844-ib 1022.315u 13.870s 17:16.81 99.9% 0+0k 0+0io 0pf+0w</div>
<div> Summary of lapw2para:</div><div> c1208-ib<span class="" style="white-space:pre"> </span> user=1016.52<span class="" style="white-space:pre"> </span> wallclock=1031.1</div><div> c1201-ib<span class="" style="white-space:pre"> </span> user=1017.36<span class="" style="white-space:pre"> </span> wallclock=1031.82</div>
<div> c1180-ib<span class="" style="white-space:pre"> </span> user=1033.06<span class="" style="white-space:pre"> </span> wallclock=1047.07</div><div> c1179-ib<span class="" style="white-space:pre"> </span> user=1037.55<span class="" style="white-space:pre"> </span> wallclock=1051.5</div>
<div> c1178-ib<span class="" style="white-space:pre"> </span> user=1019.16<span class="" style="white-space:pre"> </span> wallclock=1033.49</div><div> c1177-ib<span class="" style="white-space:pre"> </span> user=1021.88<span class="" style="white-space:pre"> </span> wallclock=1036.07</div>
<div> c1171-ib<span class="" style="white-space:pre"> </span> user=1032.42<span class="" style="white-space:pre"> </span> wallclock=1046.7</div><div> c0844-ib<span class="" style="white-space:pre"> </span> user=1022.32<span class="" style="white-space:pre"> </span> wallclock=1036.81</div>
<div>31.923u 13.526s 18:20.12 4.1%<span class="" style="white-space:pre"> </span>0+0k 0+0io 0pf+0w</div><div>> lapw2 -dn -p -c <span class="" style="white-space:pre"> </span>(19:51:10) running LAPW2 in parallel mode</div>
<div> c1208-ib 947.942u 13.364s 16:01.75 99.9% 0+0k 0+0io 0pf+0w</div><div> c1201-ib 932.766u 13.640s 15:49.22 99.7% 0+0k 0+0io 0pf+0w</div><div> c1180-ib 932.474u 13.609s 15:47.76 99.8% 0+0k 0+0io 0pf+0w</div>
<div> c1179-ib 936.171u 13.691s 15:50.33 99.9% 0+0k 0+0io 0pf+0w</div><div> c1178-ib 947.798u 13.493s 16:04.99 99.6% 0+0k 0+0io 0pf+0w</div><div> c1177-ib 947.786u 13.350s 16:04.89 99.6% 0+0k 0+0io 0pf+0w</div>
<div> c1171-ib 930.971u 13.874s 15:45.22 99.9% 0+0k 0+0io 0pf+0w</div><div> c0844-ib 950.723u 13.426s 16:04.69 99.9% 0+0k 0+0io 0pf+0w</div><div> Summary of lapw2para:</div><div> c1208-ib<span class="" style="white-space:pre"> </span> user=947.942<span class="" style="white-space:pre"> </span> wallclock=961.75</div>
<div> c1201-ib<span class="" style="white-space:pre"> </span> user=932.766<span class="" style="white-space:pre"> </span> wallclock=949.22</div><div> c1180-ib<span class="" style="white-space:pre"> </span> user=932.474<span class="" style="white-space:pre"> </span> wallclock=947.76</div>
<div> c1179-ib<span class="" style="white-space:pre"> </span> user=936.171<span class="" style="white-space:pre"> </span> wallclock=950.33</div><div> c1178-ib<span class="" style="white-space:pre"> </span> user=947.798<span class="" style="white-space:pre"> </span> wallclock=964.99</div>
<div> c1177-ib<span class="" style="white-space:pre"> </span> user=947.786<span class="" style="white-space:pre"> </span> wallclock=964.89</div><div> c1171-ib<span class="" style="white-space:pre"> </span> user=930.971<span class="" style="white-space:pre"> </span> wallclock=945.22</div>
<div> c0844-ib<span class="" style="white-space:pre"> </span> user=950.723<span class="" style="white-space:pre"> </span> wallclock=964.69</div><div>31.522u 13.879s 16:53.13 4.4%<span class="" style="white-space:pre"> </span>0+0k 0+0io 0pf+0w</div>
<div>> lcore -up<span class="" style="white-space:pre"> </span>(20:08:03) 2.993u 0.587s 0:03.75 95.2%<span class="" style="white-space:pre"> </span>0+0k 0+0io 0pf+0w</div><div>> lcore -dn<span class="" style="white-space:pre"> </span>(20:08:07) 2.843u 0.687s 0:03.66 96.1%<span class="" style="white-space:pre"> </span>0+0k 0+0io 0pf+0w</div>
<div>> mixer <span class="" style="white-space:pre"> </span>(20:08:21) 23.206u 32.513s 0:56.63 98.3%<span class="" style="white-space:pre"> </span>0+0k 0+0io 0pf+0w</div><div>:ENERGY convergence: 0 0.00001 416.9302585700000000</div>
<div>:CHARGE convergence: 0 0.0000 3.6278086</div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Oct 17, 2013 at 7:11 AM, Laurence Marks <span dir="ltr"><<a href="mailto:L-marks@northwestern.edu" target="_blank">L-marks@northwestern.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">There are so many possibilities, a few:<br>
<br>
a) If you only request 1 core/node most queuing systems (qsub/msub<br>
etc) will allocate the other cores to other jobs. You are then going<br>
to be very dependent upon what those other jobs are doing. Normal is<br>
to use all the cores on a given node.<br>
<br>
b) When you run on cluster B, in addition to a) it is going to be<br>
inefficient to run with mpi communications across nodes and it is much<br>
better to run on a given node across cores. Are you using a machines<br>
file with eight 1: nodeA lines (for instance) or one with a single 1:<br>
nodeA nodeB....? The first does not use mpi, the second does. To use<br>
mpi within a node you would use lines such as 1:node:8. Knowledge of<br>
your .machines file will help people assist you.<br>
<br>
c) The memory on those clusters is very small, whoever bought them was<br>
not thinking about large scale jobs. I look for at least 4G/core, and<br>
2G/core is barely acceptable. You are going to have to use mpi.<br>
<br>
d) All mpi is equal, but some mpi is more equal than others. Depending<br>
upon whether you have infiniband, ethernet, openmpi, impi and how<br>
everything was compiled you can see enormous differences. One thing to<br>
look at is the difference between the cpu time and wall time (both in<br>
case.dayfile and at the bottom of case.output1_*). With a good mpi<br>
setup the wall time should be 5-10% more than the cpu time; with a bad<br>
setup it can be several times it.<br>
<div><div class="h5"><br>
On Thu, Oct 17, 2013 at 8:44 AM, Yundi Quan <<a href="mailto:quanyundi@gmail.com">quanyundi@gmail.com</a>> wrote:<br>
> Hi,<br>
> I have access to two clusters as a low-level user. One cluster (cluster A)<br>
> consists of nodes with 8 core and 8 G mem per node. The other cluster<br>
> (cluster B) has 24G mem per node and each node has 14 cores or more. The<br>
> cores on cluster A are Xeon CPU E5620@2.40GHz, while the cores on cluster B<br>
> are Xeon CPU X5550@2.67GH. From the specifications (2.40GHz+12288 KB cache<br>
> vs 2.67GHz+8192 KB cache), two machines should be very close in performance.<br>
> But it does not seem to be so.<br>
><br>
> I have job with 72 atoms per unit cell. I initialized the job on cluster A<br>
> and ran it for a few iterations. Each iteration took 2 hours. Then, I moved<br>
> the job to cluster B (14 cores per node with @2.67GHz). Now it takes more<br>
> than 8 hours to finish one iteration. On both clusters, I request one core<br>
> per node and 8 nodes per job ( 8 is the number of k points). I compiled<br>
> WIEN2k_13 on cluster A without mpi. On cluster B, WIEN2k_12 was compiled by<br>
> the administrator with mpi.<br>
><br>
> What could have caused poor performance of cluster B? Is it because of MPI?<br>
><br>
> On an unrelated question. Sometimes memory would run out on cluster B which<br>
> has 24Gmem per node. Nevertheless the same job could run smoothly on cluster<br>
> A which only has 8 G per node.<br>
><br>
> Thanks.<br>
<br>
<br>
<br>
</div></div>--<br>
Professor Laurence Marks<br>
Department of Materials Science and Engineering<br>
Northwestern University<br>
<a href="http://www.numis.northwestern.edu" target="_blank">www.numis.northwestern.edu</a> <a href="tel:1-847-491-3996" value="+18474913996">1-847-491-3996</a><br>
"Research is to see what everybody else has seen, and to think what<br>
nobody else has thought"<br>
Albert Szent-Gyorgi<br>
_______________________________________________<br>
Wien mailing list<br>
<a href="mailto:Wien@zeus.theochem.tuwien.ac.at">Wien@zeus.theochem.tuwien.ac.at</a><br>
<a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien" target="_blank">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a><br>
SEARCH the MAILING-LIST at: <a href="http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html" target="_blank">http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html</a><br>
</blockquote></div><br></div>