<div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif;color:#000000">Please do "cat $WIENROOT/parallel_options", as I suspect you have an issue there. </div><div class="gmail_default" style="font-family:verdana,sans-serif;color:#000000">Do you have a "normal" mpirun or does your cluster require something different? </div><div class="gmail_default" style="font-family:verdana,sans-serif;color:#000000">Which mpirun are you using?</div><div class="gmail_default" style="font-family:verdana,sans-serif;color:#000000"><br></div>Also, I doubt you need "lapw2_vector_split:2", and you do not appear to have set the "omp_XXX" variables which are needed for <span class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(0,0,0)">recent versions</span>.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jan 31, 2023 at 10:59 AM Calum Cunningham <<a href="mailto:Calum.Cunningham@uknnl.com">Calum.Cunningham@uknnl.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg2139504228931482943">
<div lang="EN-GB" style="overflow-wrap: break-word;">
<div class="m_2139504228931482943WordSection1">
<p class="MsoNormal">Dear WIEN2k users,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">My colleagues and I are having some trouble running SCF calculations in parallel mode. I have had no issues when working in serial mode. We are using version 21.1 on a computer cluster that operates the LSF queuing system.
<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">As an example, I will explain my attempt to run a parallel execution for the TiO2 (rutile) test case. I am using the default values of RKmax, k-points, VXC, etc.
<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">The .machines file was created using a bespoke script that updates the names of the processors being used for the current job. In this case, I am using 16 cores on a single node. The .machines file is below:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New""># .machines file for Wien2k<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">#<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New""><u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">1:sqg1cintr16.bullx:16<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">granularity:1<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">extrafine:1<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">lapw0: sqg1cintr16.bullx:16
<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">dstart: sqg1cintr16.bullx:16
<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">nlvdw: sqg1cintr16.bullx:16
<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">lapw2_vector_split:2<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">After I initialise the calculation interactively via the w2web GUI (i.e. not in parallel), I attempted to execute the SCF cycle in w2web with the parallel option selected. I received the following error in STDOUT:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">LAPW0 END<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">[1] Done mpirun -np 16 /lustre/scafellpike/local/apps/intel/wien2k/21.1/lapw0_mpi lapw0.def >> .time00<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">LAPW1 END<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">[1] + Done ( cd $PWD; $t $ttt; rm -f .lock_$lockfile[$p] ) >> .time1_$loop<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">tmpmach: Subscript out of range.<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">grep: lapw2*.error: No such file or directory<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">> stop error</span><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Note that I consistently receive this “grep: lapw2*.error” error when attempting to run SCF calculations in parallel! After this, I tested each of lapw0, lapw1 and lapw2 as single programmes (in parallel) to try to fix the problem. I think
that lapw1 ran correctly, but I have given the output below just in case there is a problem here. There is, however, an obvious error when lapw2 is executed (see below).<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">starting parallel lapw1 at Tue Jan 31 15:00:07 GMT 2023<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">-> starting parallel LAPW1 jobs at Tue Jan 31 15:00:07 GMT 2023<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">running LAPW1 in parallel mode (using .machines)<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">granularity set to 1 because of nonlocal SCRATCH variable<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">1 number_of_parallel_jobs<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">[1] 46212<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">LAPW1 END<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">[1] + Done ( cd $PWD; $t $ttt; rm -f .lock_$lockfile[$p] ) >> .time1_$loop<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New""> (70) 0.011u 0.027s 0:14.52 0.2% 0+0k 0+8io 0pf+0w<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New""> Summary of lapw1para:<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New""> sqg1cintr16.bullx k= user= wallclock=<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">0.100u 0.299s 0:16.85 2.3% 0+0k 616+248io 0pf+0w<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><u></u> <u></u></p>
<p class="MsoNormal" style="margin-left:36pt">#lapw2 as a single programme (parallel):<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">running LAPW2 in parallel mode<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">tmpmach: Subscript out of range.<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">0.016u 0.043s 0:00.06 83.3% 0+0k 32+24io 0pf+0w<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-family:"Courier New"">error: command /lustre/scafellpike/local/apps/intel/wien2k/21.1/lapw2para lapw2.def failed</span><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Please let me know if you need any more information. I would particularly like to know why the errors are occurring at lapw2 (e.g. what is the “tmpmach” error?)<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Many thanks,<u></u><u></u></p>
<p class="MsoNormal">Calum Cunningham<u></u><u></u></p>
</div>
This e-mail is from the National Nuclear Laboratory Limited (NNL). This e-mail and any attachments are intended for the addressee and may also be legally privileged. If you are not the intended recipient please do not print, re-transmit, store or act in reliance
on it or any attachments. Instead, please e-mail it back to the sender and then immediately permanently delete it. National Nuclear Laboratory Limited (Company Number 3857752) Registered in England and Wales. Registered office Chadwick House, Warrington Road,
Birchwood Park, Warrington, WA3 6AE.
</div>
_______________________________________________<br>
Wien mailing list<br>
<a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a><br>
<a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien" rel="noreferrer" target="_blank">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a><br>
SEARCH the MAILING-LIST at: <a href="http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html" rel="noreferrer" target="_blank">http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html</a><br>
</div></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr">Professor Laurence Marks<br>Department of Materials Science and Engineering<br>Northwestern University<br><a href="http://www.numis.northwestern.edu" target="_blank">www.numis.northwestern.edu</a> <br>"Research is to see what everybody else has seen, and to think what nobody else has thought", Albert Szent-Györgyi</div></div>