<div dir="auto"><div>I think you are doing something wrong in your job submission. I suggest that you talk to your sysadmin, as there are too many ways for your calculations to have gone wrong. It may take weeks or more of people on the list guessing.<div dir="auto"><br></div><div dir="auto">It should be possible to assign nodes interactively and have them available in .machines. Your response that the simple commands fail with "pbsssh: command not found" is very odd. The command "x lapw0 -p" is a very basic one, and if this fails for multiple cores something is very wrong. <br><br><div data-smartmail="gmail_signature" dir="auto">---<br>Prof Laurence Marks<br>"Research is to see what everyone else has seen, and to think what nobody else has thought", Albert Szent-Gyorgi<br><a href="http://www.numis.northwestern.edu">www.numis.northwestern.edu</a></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Oct 24, 2020, 15:30 Christian Søndergaard Pedersen <<a href="mailto:chrsop@dtu.dk">chrsop@dtu.dk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div id="m_8929786822793361747divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif" dir="ltr">
<div id="m_8929786822793361747divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p>Hello Gavin</p>
<p><br>
</p>
<p>Thanks for your reply, and apologies for my tardiness. </p>
<p><br>
</p>
<p>[1] All my calculations are run in MPI-parallel on our HPC cluster. I cannot execute any 'x lapw[0,1,2] -p' command in the terminal (on the cluster login node); this results in 'pbsssh: command not found'. However, submitting via the SLURM workload manager
works fine. In all my submit scripts, I specify 'setenv SCRATCH /scratch/$USER', which is the proper location of scratch storage on our HPC cluster.</p>
<p><br>
</p>
<p>[2] Without having tried your example for diamond, I can report that 'run_lapw -p' followed by 'x qtl -p -telnes' works without problems for a single cell of Vanadium dioxide. However, for other systems I get the error I specified. The other systems (1)
are larger, and (2) use two CPU's instead of a single CPU (.machines file are modified suitably).</p>
<p>Checking the qtl.def file for the calculation that _did_ work, I can see that the line specifying
<span>'/scratch/chrsop/VO2.vectordn'</span> is _also_ present here, so this is not to blame. This leaves me baffled as to what the error can be - as far as I can tell, I am trying to perform the exact same calculation for different systems. I thought maybe
insufficient scratch storage could be to blame, but this would most likely show up in the 'run_lapw' cycles (I believe).</p>
<p><br>
</p>
<p>[3] I am posting here the difference between qtlpara and lapw2para:</p>
<p></p>
<div><span style="font-family:"Courier New",monospace"> </span><span style="font-family:"Courier New",monospace">$ grep "single" $WIENROOT/qtlpara_lapw</span><br>
<span style="font-family:"Courier New",monospace"> testinput .processes single</span><br>
<span style="font-family:"Courier New",monospace"> $ grep "single" $WIENROOT/lapw2para_lapw</span><br>
<span style="font-family:"Courier New",monospace"> testinput .processes single</span><br>
<span style="font-family:"Courier New",monospace"> single:</span><br>
<span style="font-family:"Courier New",monospace"> echo "running in single mode"</span></div>
<div><br>
</div>
<div>... if this is wrong, I kindly request advice on how to fix it, so I can pass it on to our software maintenance guy. If there's anything else I can try please let me know.</div>
<div><br>
</div>
<div>Best regards<br>
Christian<br>
</div>
<br>
<p></p>
<p> <br>
</p>
</div>
<hr style="display:inline-block;width:98%">
<div id="m_8929786822793361747divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>Fra:</b> Wien <<a href="mailto:wien-bounces@zeus.theochem.tuwien.ac.at" target="_blank" rel="noreferrer">wien-bounces@zeus.theochem.tuwien.ac.at</a>> på vegne af Gavin Abo <<a href="mailto:gsabo@crimson.ua.edu" target="_blank" rel="noreferrer">gsabo@crimson.ua.edu</a>><br>
<b>Sendt:</b> 21. oktober 2020 07:02:01<br>
<b>Til:</b> <a href="mailto:wien@zeus.theochem.tuwien.ac.at" target="_blank" rel="noreferrer">wien@zeus.theochem.tuwien.ac.at</a><br>
<b>Emne:</b> Re: [Wien] qtl: error reading parallel vectors</font>
<div> </div>
</div>
<div>
<p>I'm not sure about the physics of the following WIEN2k 19.2 parallel calculation (with all patches at [1] applied), but mechanically the "x qtl -p -telnes" seems to have run without error.</p>
<p><br>
</p>
<p>I typically have SCRATCH in my .bashrc set to "./" but used another location "/home/username/wiendata/scratch" as seen below. Does a simple k-point parallel calculation like the one below work on your system? I haven't tried mpi parallel yet. On the other
hand, I have noticed a possible issue that if one forgets to setup a .machines file and tries to run a parallel calculation that qtlpara_lapw seems to fail switching over to the serial calculation mode as shown under [2] below. If one compares for example
lapw2para_lapw and qtlpara_lapw, as illustrated by [3] below, the qtlpara_lapw may be missing some additional code that could be needed to get that to work.<br>
</p>
<p><br>
</p>
<p>username@computername:~/wiendata/diamond$ grep SCRATCH ~/.bashrc<br>
export SCRATCH=/home/username/wiendata/scratch<br>
username@computername:~/wiendata/diamond$ ls<br>
diamond.struct<br>
username@computername:~/wiendata/diamond$ init_lapw -b<br>
...<br>
init_lapw finished ok<br>
username@computername:~/wiendata/diamond$ cat .machines<br>
1:localhost<br>
1:localhost<br>
granularity:1<br>
extrafine:1<br>
username@computername:~/wiendata/diamond$ run_lapw -p<br>
...<br>
in cycle 11 ETEST: .0001457550000000 CTEST: .0033029<br>
hup: Command not found.<br>
STOP LAPW0 END<br>
STOP LAPW1 END<br>
STOP LAPW1 END<br>
STOP LAPW2 - FERMI; weights written<br>
STOP LAPW2 END<br>
STOP LAPW2 END<br>
STOP SUMPARA END<br>
STOP CORE END<br>
STOP MIXER END<br>
ec cc and fc_conv 1 1 1<br>
<br>
> stop<br>
username@computername:~/wiendata/diamond$ cp $WIENROOT/SRC_templates/case.innes diamond.innes<br>
username@computername:~/wiendata/diamond$ x qtl -p -telnes<br>
running QTL in parallel mode<br>
calculating QTL's from parallel vectors<br>
STOP QTL END<br>
6.4u 0.1s 0:06.59 100.0% 0+0k 0+8024io 0pf+0w<br>
username@computername:~/wiendata/diamond$ cat diamond.inq<br>
0 2.20000000000000000000<br>
1<br>
1 99 1 0<br>
4 0 1 2 3<br>
username@computername:~/wiendata/diamond$ x telnes3<br>
STOP TELNES3 DONE<br>
3.3u 0.0s 0:03.39 99.7% 0+0k 0+96io 0pf+0w<br>
</p>
<p><br>
</p>
<p>[1] <a href="https://urldefense.com/v3/__https://github.com/gsabo/WIEN2k-Patches/tree/master/19.2__;!!Dq0X2DkFhyF93HkjWTBQKhk!HnO7YK88PsHaa54xa5ASPZYPlOBOjdZoteJKx-B9H6XjTztBItejXIR8aRUf7sogxXHnPw$" id="m_8929786822793361747LPlnk993820" target="_blank" rel="noreferrer">
https://github.com/gsabo/WIEN2k-Patches/tree/master/19.2</a></p>
<div id="m_8929786822793361747LPBorder_GT_16035703620450.658529909935345" style="margin-bottom:20px;overflow:auto;width:100%;text-indent:0px">
<table id="m_8929786822793361747LPContainer_16035703620390.7134542849958105" style="width:90%;background-color:rgb(255,255,255);overflow:auto;padding-top:20px;padding-bottom:20px;margin-top:20px;border-top:1px dotted rgb(200,200,200);border-bottom:1px dotted rgb(200,200,200)" cellspacing="0">
<tbody>
<tr style="border-spacing:0px" valign="top">
<td id="m_8929786822793361747ImageCell_16035703620400.7188831811826296" style="width:250px;display:table-cell;padding-right:20px" colspan="1">
<div id="m_8929786822793361747LPImageContainer_16035703620400.05900536828780922" style="background-color:rgb(255,255,255);height:250px;margin:auto;display:table;width:250px">
<a id="m_8929786822793361747LPImageAnchor_16035703620420.633020516273475" style="display:table-cell;text-align:center" href="https://urldefense.com/v3/__https://github.com/gsabo/WIEN2k-Patches/tree/master/19.2__;!!Dq0X2DkFhyF93HkjWTBQKhk!HnO7YK88PsHaa54xa5ASPZYPlOBOjdZoteJKx-B9H6XjTztBItejXIR8aRUf7sogxXHnPw$" target="_blank" rel="noreferrer"><img style="display:inline-block;max-width:250px;max-height:250px;height:250px;width:250px;border-width:0px;vertical-align:bottom" id="m_8929786822793361747LPThumbnailImageID_16035703620420.40860880472157823" width="250" height="250" src="https://avatars0.githubusercontent.com/u/6389916?s=400&v=4"></a></div>
</td>
<td id="m_8929786822793361747TextCell_16035703620420.7389187826906752" style="vertical-align:top;padding:0px;display:table-cell" colspan="2">
<div id="m_8929786822793361747LPRemovePreviewContainer_16035703620420.23397685413583114"></div>
<div id="m_8929786822793361747LPTitle_16035703620430.5035598590199127" style="color:rgb(0,120,215);font-weight:400;font-size:21px;font-family:"wf_segoe-ui_light","Segoe UI Light","Segoe WP Light","Segoe UI","Segoe WP",Tahoma,Arial,sans-serif;line-height:21px">
<a id="m_8929786822793361747LPUrlAnchor_16035703620430.15080431579336695" style="text-decoration:none" href="https://urldefense.com/v3/__https://github.com/gsabo/WIEN2k-Patches/tree/master/19.2__;!!Dq0X2DkFhyF93HkjWTBQKhk!HnO7YK88PsHaa54xa5ASPZYPlOBOjdZoteJKx-B9H6XjTztBItejXIR8aRUf7sogxXHnPw$" target="_blank" rel="noreferrer">WIEN2k-Patches/19.2 at master · gsabo/WIEN2k-Patches · GitHub</a></div>
<div id="m_8929786822793361747LPMetadata_16035703620430.39140898497152976" style="margin:10px 0px 16px;color:rgb(102,102,102);font-weight:400;font-family:"wf_segoe-ui_normal","Segoe UI","Segoe WP",Tahoma,Arial,sans-serif;font-size:14px;line-height:14px">
<a href="http://github.com" target="_blank" rel="noreferrer">github.com</a></div>
<div id="m_8929786822793361747LPDescription_16035703620440.5382112202329979" style="display:block;color:rgb(102,102,102);font-weight:400;font-family:"wf_segoe-ui_normal","Segoe UI","Segoe WP",Tahoma,Arial,sans-serif;font-size:14px;line-height:20px;max-height:100px;overflow:hidden">
Contribute to gsabo/WIEN2k-Patches development by creating an account on GitHub.</div>
</td>
</tr>
</tbody>
</table>
</div>
<br>
<p></p>
<p><br>
</p>
<p>[2] Error when qtlpara_lapw tries to switch to single mode during "x qtl -p -telnes":<br>
</p>
<p><br>
</p>
<p>username@computername:~/wiendata/diamond$ cat .machine<br>
cat: .machine: No such file or directory<br>
username@computername:~/wiendata/diamond$ run_lapw -p<br>
...<br>
in cycle 11 ETEST: .0001457550000000 CTEST: .0033029<br>
hup: Command not found.<br>
STOP LAPW0 END<br>
STOP LAPW1 END<br>
STOP LAPW2 END<br>
STOP CORE END<br>
STOP MIXER END<br>
ec cc and fc_conv 1 1 1<br>
<br>
> stop<br>
username@computername:~/wiendata/diamond$ cp $WIENROOT/SRC_templates/case.innes diamond.innes<br>
username@computername:~/wiendata/diamond$ x qtl -p -telnes<br>
single: label not found.<br>
0.0u 0.0s 0:00.01 0.0% 0+0k 0+0io 0pf+0w<br>
error: command /home/username/WIEN2k/qtlpara qtl.def failed<br>
</p>
<p><br>
</p>
<p>[3] Grep difference between qtlpara_lapw and lapw2para_lapw:<br>
</p>
<p><br>
</p>
<p>username@computername:~/wiendata/diamond$ grep "single" $WIENROOT/qtlpara_lapw<br>
testinput .processes single<br>
username@computername:~/wiendata/diamond$ grep "single" $WIENROOT/lapw2para_lapw<br>
testinput .processes single<br>
single:<br>
echo "running in single mode"<br>
</p>
<p><br>
</p>
<div>On 10/20/2020 12:24 PM, Christian Søndergaard Pedersen wrote:<br>
</div>
<blockquote type="cite"></blockquote>
</div>
</div>
</div>
_______________________________________________<br>
Wien mailing list<br>
<a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank" rel="noreferrer">Wien@zeus.theochem.tuwien.ac.at</a><br>
<a href="https://urldefense.com/v3/__http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien__;!!Dq0X2DkFhyF93HkjWTBQKhk!HnO7YK88PsHaa54xa5ASPZYPlOBOjdZoteJKx-B9H6XjTztBItejXIR8aRUf7sqXggOxWw$" rel="noreferrer noreferrer" target="_blank">https://urldefense.com/v3/__http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien__;!!Dq0X2DkFhyF93HkjWTBQKhk!HnO7YK88PsHaa54xa5ASPZYPlOBOjdZoteJKx-B9H6XjTztBItejXIR8aRUf7sqXggOxWw$</a> <br>
SEARCH the MAILING-LIST at: <a href="https://urldefense.com/v3/__http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html__;!!Dq0X2DkFhyF93HkjWTBQKhk!HnO7YK88PsHaa54xa5ASPZYPlOBOjdZoteJKx-B9H6XjTztBItejXIR8aRUf7so6BN45lw$" rel="noreferrer noreferrer" target="_blank">https://urldefense.com/v3/__http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html__;!!Dq0X2DkFhyF93HkjWTBQKhk!HnO7YK88PsHaa54xa5ASPZYPlOBOjdZoteJKx-B9H6XjTztBItejXIR8aRUf7so6BN45lw$</a> <br>
</blockquote></div></div></div>