<div dir="ltr"><div dir="ltr"><div dir="ltr">Memory should be fine since I have chosen an idle node to do the tests and I have tested a few omp_global values, it still performed badly. So, it should be a file/network issue as Prof. Blaha suggested since I have indeed set the SCRATCH directory as ./. Anyway, thank you for your suggestions.</div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Laurence Marks <<a href="mailto:laurence.marks@gmail.com">laurence.marks@gmail.com</a>> 于2020年12月28日周一 下午7:13写道：<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">While the server is probably the issue, two other things to check:<div dir="auto">a) lapw2 can need more memory, so swapping issues are not impossible.</div><div dir="auto">b) How have you set omp_lapw2 ?<br><br><div dir="auto">_____<br>Professor Laurence Marks<br>"Research is to see what everybody else has seen, and to think what nobody else has thought", Albert Szent-Gyorgi<br><a href="http://www.numis.northwestern.edu" target="_blank">www.numis.northwestern.edu</a></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Dec 28, 2020, 03:23 Fan <<a href="mailto:fxinwei123@gmail.com" target="_blank">fxinwei123@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thank you for your quick response. I believe that is the key to the problem. So with larger vector files, the lapw2 will run slower in my situation. That makes sense.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Peter Blaha <<a href="mailto:pblaha@theochem.tuwien.ac.at" rel="noreferrer" target="_blank">pblaha@theochem.tuwien.ac.at</a>> 于2020年12月28日周一 下午4:38写道：<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">This points to a fileserver/network problem on your cluster.<br>
<br>
On a big cluster one usually has a home-directory on a fileserver drive, <br>
which is mounted over a network (eg. NFS, HPFS,..) on all nodes. While <br>
this is very convenient, it is on many systems also a huge bottleneck. <br>
Either the fileserver breaks down due to too many simultaneously <br>
requests or the network does not have sufficient band width.<br>
<br>
You probably have set the SCRATCH variable to your working directory <br>
"./", which means that the large case.vector* files are stored in your <br>
working directory, leading to the bad performance of lapw2.<br>
<br>
On most clusters there is a local scratch or tmp directory (files exist <br>
only on this node) and when setting your SCRATCH variable to this, the <br>
slow lapw2 should be gone.<br>
<br>
In addition, I see that also lapw1 gets only 90% of a core. I recommend <br>
setting OMP_NUM_THREADS=2 and spanning only 4 k-parallel jobs in <br>
parallel (I assume you have a node with 8 cores).<br>
<br>
<br>
On 12/28/20 9:14 AM, Fan wrote:<br>
> Dear wien2k users,<br>
> <br>
> I am trying to run wien2k on clusters, but I encountered a very strange <br>
> issue that the performance of lapw2 is very inefficient. For example,<br>
> <br>
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
> > lapw1 -dn -p (15:27:45) starting parallel lapw1 at Mon Dec <br>
> 28 15:27:46 CST 2020<br>
> -> starting parallel LAPW1 jobs at Mon Dec 28 15:27:46 CST 2020<br>
> running LAPW1 in parallel mode (using .machines.help)<br>
> 8 number_of_parallel_jobs<br>
> f02n10(58) 178.212u 3.134s 3:15.91 92.57% 0+0k 0+0io 0pf+0w<br>
> f02n10(58) 180.210u 3.011s 3:19.46 91.86% 0+0k 0+0io 0pf+0w<br>
> f02n10(58) 183.239u 3.019s 3:22.73 91.87% 0+0k 0+0io 0pf+0w<br>
> f02n10(57) 181.113u 2.884s 3:20.20 91.90% 0+0k 0+0io 0pf+0w<br>
> f02n10(57) 178.433u 2.965s 3:18.85 91.22% 0+0k 0+0io 0pf+0w<br>
> f02n10(57) 151.420u 2.756s 2:48.26 91.63% 0+0k 0+0io 0pf+0w<br>
> f02n10(57) 183.799u 3.065s 3:22.58 92.24% 0+0k 0+0io 0pf+0w<br>
> f02n10(57) 185.867u 3.109s 3:27.50 91.07% 0+0k 0+0io 0pf+0w<br>
> Summary of lapw1para:<br>
> f02n10 k=459 user=1422.29 wallclock=1575.49<br>
> 1.034u 1.698s 3:30.15 1.2% 0+0k 472+128io 2pf+0w<br>
> > lapw2 -up -p (15:31:16) running LAPW2 in parallel mode<br>
> f02n10 37.680u 1.597s 6:32.94 10.00% 0+0k 0+0io 0pf+0w<br>
> f02n10 38.841u 1.682s 7:07.57 9.48% 0+0k 0+0io 0pf+0w<br>
> f02n10 38.611u 1.727s 6:51.53 9.80% 0+0k 0+0io 0pf+0w<br>
> f02n10 38.715u 1.728s 6:48.76 9.89% 0+0k 0+0io 0pf+0w<br>
> f02n10 37.847u 1.639s 7:01.08 9.38% 0+0k 0+0io 0pf+0w<br>
> f02n10 38.170u 1.709s 6:45.01 9.85% 0+0k 0+0io 0pf+0w<br>
> f02n10 39.261u 1.727s 7:01.11 9.73% 0+0k 0+0io 0pf+0w<br>
> f02n10 39.772u 1.765s 7:04.40 9.79% 0+0k 0+0io 0pf+0w<br>
> <br>
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
> As you can see the cpu utilizations are much lower than that of lapw1, <br>
> which made lawp2 more time-consuming than lapw1. MPI parallelization <br>
> performed even worse.<br>
> <br>
> More strangely, it seems to be case-dependent. For TiC and some other <br>
> systems, it works fine. I also tried other nodes, the problem still <br>
> persists, however in my local workstation everything went well.<br>
> <br>
> The version of wien2k is 19.2 compiled with intel icc and ifort without <br>
> any errors. MPI, FFTW, and ELPA are all available.<br>
> <br>
> Any suggestion will be appreciated.<br>
> <br>
> Fan<br>
> <br>
> <br>
> <br>
> <br>
> <br>
> _______________________________________________<br>
> Wien mailing list<br>
> <a href="mailto:Wien@zeus.theochem.tuwien.ac.at" rel="noreferrer" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a><br>
> <a href="https://urldefense.com/v3/__http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien__;!!Dq0X2DkFhyF93HkjWTBQKhk!EjV3JB-TAlZJGOYIcsqasIMWKmgqDXUbL1tizifidFHUioB3wf4Tl84i7MNpF0O2-yJfew$" rel="noreferrer noreferrer" target="_blank">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a><br>
> SEARCH the MAILING-LIST at: <a href="https://urldefense.com/v3/__http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html__;!!Dq0X2DkFhyF93HkjWTBQKhk!EjV3JB-TAlZJGOYIcsqasIMWKmgqDXUbL1tizifidFHUioB3wf4Tl84i7MNpF0OxfQ5bgA$" rel="noreferrer noreferrer" target="_blank">http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html</a><br>
> <br>
<br>
-- <br>
<br>
P.Blaha<br>
--------------------------------------------------------------------------<br>
Peter BLAHA, Inst.f. Materials Chemistry, TU Vienna, A-1060 Vienna<br>
Phone: +43-1-58801-165300 FAX: +43-1-58801-165982<br>
Email: <a href="mailto:blaha@theochem.tuwien.ac.at" rel="noreferrer" target="_blank">blaha@theochem.tuwien.ac.at</a> WIEN2k: <a href="https://urldefense.com/v3/__http://www.wien2k.at__;!!Dq0X2DkFhyF93HkjWTBQKhk!EjV3JB-TAlZJGOYIcsqasIMWKmgqDXUbL1tizifidFHUioB3wf4Tl84i7MNpF0NUbjv7Xw$" rel="noreferrer noreferrer" target="_blank">http://www.wien2k.at</a><br>
WWW: <a href="https://urldefense.com/v3/__http://www.imc.tuwien.ac.at/TC_Blaha__;!!Dq0X2DkFhyF93HkjWTBQKhk!EjV3JB-TAlZJGOYIcsqasIMWKmgqDXUbL1tizifidFHUioB3wf4Tl84i7MNpF0PkvhVHaQ$" rel="noreferrer noreferrer" target="_blank">http://www.imc.tuwien.ac.at/TC_Blaha</a><br>
--------------------------------------------------------------------------<br>
_______________________________________________<br>
Wien mailing list<br>
<a href="mailto:Wien@zeus.theochem.tuwien.ac.at" rel="noreferrer" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a><br>
<a href="https://urldefense.com/v3/__http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien__;!!Dq0X2DkFhyF93HkjWTBQKhk!EjV3JB-TAlZJGOYIcsqasIMWKmgqDXUbL1tizifidFHUioB3wf4Tl84i7MNpF0O2-yJfew$" rel="noreferrer noreferrer" target="_blank">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a><br>
SEARCH the MAILING-LIST at: <a href="https://urldefense.com/v3/__http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html__;!!Dq0X2DkFhyF93HkjWTBQKhk!EjV3JB-TAlZJGOYIcsqasIMWKmgqDXUbL1tizifidFHUioB3wf4Tl84i7MNpF0OxfQ5bgA$" rel="noreferrer noreferrer" target="_blank">http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html</a><br>
</blockquote></div>
_______________________________________________<br>
Wien mailing list<br>
<a href="mailto:Wien@zeus.theochem.tuwien.ac.at" rel="noreferrer" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a><br>
<a href="https://urldefense.com/v3/__http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien__;!!Dq0X2DkFhyF93HkjWTBQKhk!EjV3JB-TAlZJGOYIcsqasIMWKmgqDXUbL1tizifidFHUioB3wf4Tl84i7MNpF0O2-yJfew$" rel="noreferrer noreferrer" target="_blank">https://urldefense.com/v3/__http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien__;!!Dq0X2DkFhyF93HkjWTBQKhk!EjV3JB-TAlZJGOYIcsqasIMWKmgqDXUbL1tizifidFHUioB3wf4Tl84i7MNpF0O2-yJfew$</a> <br>
SEARCH the MAILING-LIST at: <a href="https://urldefense.com/v3/__http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html__;!!Dq0X2DkFhyF93HkjWTBQKhk!EjV3JB-TAlZJGOYIcsqasIMWKmgqDXUbL1tizifidFHUioB3wf4Tl84i7MNpF0OxfQ5bgA$" rel="noreferrer noreferrer" target="_blank">https://urldefense.com/v3/__http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html__;!!Dq0X2DkFhyF93HkjWTBQKhk!EjV3JB-TAlZJGOYIcsqasIMWKmgqDXUbL1tizifidFHUioB3wf4Tl84i7MNpF0OxfQ5bgA$</a> <br>
</blockquote></div>
_______________________________________________<br>
Wien mailing list<br>
<a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a><br>
<a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien" rel="noreferrer" target="_blank">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a><br>
SEARCH the MAILING-LIST at: <a href="http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html" rel="noreferrer" target="_blank">http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html</a><br>
</blockquote></div>