<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>/var/spool/torque/mom_priv/jobs/44.milkbar-computer.kage.SC: line
12: run_lapw: command not found<br>
</p>
<p><font face="Times New Roman">Perhaps the environmental variables
need pushed out to all nodes, you might try adding the line #PBS
-V [1,2] to your job submission script.<br>
<br>
[1] <a class="moz-txt-link-freetext" href="http://www.nics.tennessee.edu/node/387">http://www.nics.tennessee.edu/node/387</a><br>
[2]
<a class="moz-txt-link-freetext" href="http://www.open-mpi.org/community/lists/users/2008/10/6982.php">http://www.open-mpi.org/community/lists/users/2008/10/6982.php</a></font><br>
</p>
<div class="moz-cite-prefix">On 6/28/2017 11:49 PM, Yoji Kobayashi
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:D2553F99-8803-4EF6-8C49-02BEA2F2B38B@scl.kyoto-u.ac.jp">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
Dear Users,
<div class=""><br class="">
</div>
<div class="">I have a some questions/problems regarding
parallelization and PBS. </div>
<div class="">I’m not sure if I’m really running parallel vs.
serial, and my PBS script isn’t working.</div>
<div class=""><br class="">
</div>
<div class="">===</div>
<div class="">My system info:</div>
<div class="">Intel Xeon CPU E5-2630 v2 @2.6 GHz, 24 CPUS</div>
<div class="">Memory: 32GB</div>
<div class="">Running Wien2k_13, on Ubuntu 14.04.03</div>
<div class="">File system: ext4</div>
<div class="">(This is considered a single node with 24
processors?)</div>
<div class="">===</div>
<div class="">My first question is, am I really running a parallel
calculation in a meaningful way?</div>
<div class=""><br class="">
</div>
<div class="">What I try:</div>
<div class="">In w2web, a serial calculation (SCF only) for the
TiC example (500 k points) takes about 25 sec. to converge.</div>
<div class="">I do the same calculation (starting with a new case)
but setting parallelization in w2web, with slightly different <font
class="" face="Courier">.machine</font> files for each case:</div>
<div class=""><br class="">
</div>
<div class="">Case 1:</div>
<div class=""><font class="" face="Courier">1:localhost</font></div>
<div class=""><br class="">
</div>
<div class="">Case 2 (i.e. 20 lines of below):</div>
<div class="">
<div class=""><font class="" face="Courier">1:localhost</font></div>
<div class=""><font class="" face="Courier">1:localhost</font></div>
<div class=""><font class="" face="Courier">…</font></div>
<div class=""><font class="" face="Courier">1:localhost</font></div>
<div class=""><font class="" face="Courier">1:localhost</font></div>
</div>
<div class=""><font class="" face="Courier"><br class="">
</font></div>
<div class="">Case 3</div>
<div class=""><font class="" face="Courier">1:localhost:20</font></div>
<div class=""><br class="">
</div>
<div class="">(no lines referring to granularity, etc for now)</div>
<div class=""><br class="">
</div>
<div class="">What I get:</div>
<div class="">Case 1 computes in about 54 sec;</div>
<div class="">Case 2 computes in 1min23 sec.;</div>
<div class="">Case 3 gives an error in running<font class=""
face="Courier"> lapw2</font>, see the<font class=""
face="Courier"> dayfile</font> below:</div>
<div class=""><span style="font-family: Courier, fixed;" class="">-----</span></div>
<div class=""><span style="font-family: Courier, fixed;" class="">Calculating
YK-016-TiC in /home/milkbar/Yoji/YK-016-TiC</span></div>
<div class="">
<pre style="font-family: Courier, fixed;" class="">on milkbar-computer with PID 18077
using WIEN2k_13.1 (Release 17/6/2013) in /home/milkbar/WIEN2k_13
start (2017年 6月 29日 木曜日 14:23:39 JST) with lapw0 (40/99 to go)
cycle 1 (2017年 6月 29日 木曜日 14:23:39 JST) (40/99 to go)
> lapw0 -p (14:23:39) starting parallel lapw0 at 2017年 6月 29日 木曜日 14:23:39 JST
-------- .machine0 : processors
running lapw0 in single mode
1.7u 0.0s 0:01.84 98.3% 0+0k 16+440io 0pf+0w
> lapw1 -p (14:23:41) starting parallel lapw1 at 2017年 6月 29日 木曜日 14:23:41 JST
-> starting parallel LAPW1 jobs at 2017年 6月 29日 木曜日 14:23:41 JST
running LAPW1 in parallel mode (using .machines)
1 number_of_parallel_jobs
localhost localhost localhost localhost localhost localhost localhost localhost localhost localhost localhost localhost localhost localhost localhost localhost localhost localhost localhost localhost(20) 20 total processes failed to start
0.0u 0.0s 0:00.20 10.0% 0+0k 8080+8io 23pf+0w
Summary of lapw1para:
localhost k=0 user=0 wallclock=0
0.0u 0.0s 0:02.10 0.9% 0+0k 8208+216io 24pf+0w
> lapw2 -p (14:23:43) running LAPW2 in parallel mode
** LAPW2 crashed!
0.0u 0.0s 0:00.07 28.5% 0+0k 32+104io 0pf+0w
error: command /home/milkbar/WIEN2k_13/lapw2para lapw2.def failed
> stop error</pre>
<pre style="font-family: Courier, fixed;" class="">------</pre>
<pre style="font-family: Courier, fixed;" class="">Is my “serial” calculation actually processed over 24 CPUs already, so this is why it is faster than Case 2? Or am I doing something wrong? Why does Case 3 crash? </pre>
</div>
<div class=""><br class="">
</div>
<div class="">====</div>
<div class="">My second question is about PBS.</div>
<div class="">I installed torque PBS, and created a queue:</div>
<span class="">
<div class=""><font class="" face="Courier"><br class="">
</font></div>
<div class=""><font class="" face="Courier"># create default
queue</font></div>
<div class=""><font class="" face="Courier"> qmgr -c 'create
queue batch'</font></div>
<div class=""><font class="" face="Courier"> qmgr -c 'set queue
batch queue_type = execution'</font></div>
<div class=""><font class="" face="Courier"> qmgr -c 'set queue
batch started = true'</font></div>
<div class=""><font class="" face="Courier"> qmgr -c 'set queue
batch enabled = true'</font></div>
<div class=""><font class="" face="Courier"> qmgr -c 'set queue
batch resources_default.walltime = 1:00:00'</font></div>
<div class=""><font class="" face="Courier"> qmgr -c 'set queue
batch </font><span style="font-family: Courier;" class="">resources_default.nodes</span><span
style="font-family: Courier;" class=""> = 1'</span></div>
<div class=""><font class="" face="Courier"> qmgr -c 'set server
default_queue = batch’</font></div>
<div class=""><span class=""><br class="">
</span></div>
and followed other instructions on</span>
<div class=""><span class=""><a
href="https://jabriffa.wordpress.com/2015/02/11/installing-torquepbs-job-scheduler-on-ubuntu-14-04-lts/"
class="" moz-do-not-send="true">https://jabriffa.wordpress.com/2015/02/11/installing-torquepbs-job-scheduler-on-ubuntu-14-04-lts/</a><br
class="">
<br class="">
</span>The PBS system seems to work since I can submit very
simple scripts and see them on qstat. My problem is that when I
try to submit a serial wien2k job via PBS, it gives me an error
(ultimately of course I’d like to submit them as parallel, but
because of the ambiguity above I’ve kept it to serial) . Here's
the PBS script and error message:<span class=""><br class="">
</span>
<div class=""><br class="">
</div>
<div class="">
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class="">
<div style="margin: 0px; line-height: normal;" class=""><font
class="" face="Courier"> #!/bin/tcsh</font></div>
<div style="margin: 0px; line-height: normal;" class=""><font
class="" face="Courier"> ##PBS -A your_allocation</font></div>
<div style="margin: 0px; line-height: normal;" class=""><font
class="" face="Courier"> # specify the allocation.
Change it to your allocation</font></div>
<div style="margin: 0px; line-height: normal;" class=""><font
class="" face="Courier"> #PBS -q batch</font></div>
<div style="margin: 0px; line-height: normal;" class=""><font
class="" face="Courier"> #PBS -l nodes=1:ppn=20</font></div>
<div style="margin: 0px; line-height: normal;" class=""><font
class="" face="Courier"> #PBS -l walltime=1:00:00</font></div>
<div style="margin: 0px; line-height: normal;" class=""><font
class="" face="Courier"> #PBS -o wien2k_output</font></div>
<div style="margin: 0px; line-height: normal;" class=""><font
class="" face="Courier"> #PBS -j oe</font></div>
<div style="margin: 0px; line-height: normal;" class=""><font
class="" face="Courier"> #PBS -N wien2k_test</font></div>
<div style="margin: 0px; line-height: normal;" class=""><font
class="" face="Courier"> cd $PBS_O_WORKDIR</font></div>
<div style="margin: 0px; line-height: normal;" class=""><font
class="" face="Courier"> echo hello</font></div>
<div style="margin: 0px; line-height: normal;" class=""><font
class="" face="Courier"> run_lapw -i 40 -ec .0001 -I</font></div>
</div>
</div>
<div class=""><br class="">
</div>
<div class="">Error message (contents of <font class=""
face="Courier">wien2k_output</font>):</div>
<div class="">
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures;"
class=""><font class="" face="Courier">hello</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier">/var/spool/torque/mom_priv/jobs/44.milkbar-computer.kage.SC:
line 12: run_lapw: command not found</font></span></div>
</div>
<div style="margin: 0px; font-size: 11px; line-height: normal;
font-family: Menlo; background-color: rgb(255, 255, 255);"
class=""><span style="font-variant-ligatures:
no-common-ligatures" class=""><br class="">
</span></div>
<div class=""><span style="background-color: rgb(255, 255,
255);" class="">The job is listed as complete in <font
class="" face="Courier">qs</font>tat, and the “hello” is
written into the<font class="" face="Courier"> wien2k_output</font>
file. Changing the <font class="" face="Courier">cd
$PBS_O_WORKDIR</font> to the path for the current case
hasn’t changed anything. </span>I can run <span
style="background-color: rgb(255, 255, 255);" class=""><font
class="" face="Courier">run_lapw</font></span><span
style="font-family: Menlo; font-size: 11px;
background-color: rgb(255, 255, 255);" class=""> </span>from
the command line fine, though. Also, what do I write for
allocation? (I commented it out, as I see other PBS scripts
don’t always have this.)</div>
<div class=""><br class="">
</div>
<div class="">I’ve also tried the parallel case, with the
following PBS script. I set up the <font class=""
face="Courier">.structure</font> file and do the
initialization with w2web. I leave the “parallel calculation”
option unchecked when setting up the case file in w2web.</div>
<div class=""><font class="" face="Courier"><br class="">
</font></div>
<div class="">
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier"> #!/bin/tcsh</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier"> ##PBS -A
your_allocation</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier"> #PBS -q batch</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier"> #PBS -l
nodes=1:ppn=20</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier"> #PBS -l
walltime=1:00:00</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier"> #</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier"> #PBS -o
wien2k_output</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier"> #PBS -j oe</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier"> #PBS -N
wien2k_test</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier"> cd $PBS_O_WORKDIR</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier"> #</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier"> #cat $PBS_NODEFILE
|cut -c1-6 >.machines_currentdd</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier"> #set aa=`wc
.machines_current`</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier"> #echo '#' >
.machines</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier"> #</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier"> ##example for
k-point parallel lapw1/2</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier"> set i=1</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier"> <span class="Apple-tab-span" style="white-space:pre"> </span>while
($i <= $aa[1] )</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier"> <span class="Apple-tab-span" style="white-space:pre"> </span>echo
-n '1:' >>.machines</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier"> <span class="Apple-tab-span" style="white-space:pre"> </span>head
-$i .machines_current |tail -1 >> .machines</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier"> <span class="Apple-tab-span" style="white-space:pre"> </span>@
i ++</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier"> end</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier">echo
'granularity:1' >>.machines</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier">echo 'extrafine:1'
>>.machines</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier"> #</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier"> #define here your
Wien2k command</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""><font class="" face="Courier"> run_lapw -p -i 40
-ec .0001 -I</font></span></div>
</div>
<div class=""><font class="" face="Courier"><br class="">
</font></div>
<div class="">When I submit this job via <font class=""
face="Courier">qsub</font>, again the job is immediately
listed as complete in <font class="" face="Courier">qstat</font>,
and I get the following error message in <font class=""
face="Courier">wien2k_output</font>:</div>
<div class=""><br class="">
</div>
<div class="">
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures;"
class=""><font class="" face="Courier">milkbar@milkbar-computer:~/Yoji/YK-017-TiC$
cat wien2k_output</font></span></div>
<div style="margin: 0px; line-height: normal;
background-color: rgb(255, 255, 255);" class=""><span
style="font-variant-ligatures: no-common-ligatures;"
class=""><font class="" face="Courier">/var/spool/torque/mom_priv/jobs/45.milkbar-computer.kage.SC:
line 28: syntax error: unexpected end of file</font></span></div>
</div>
<div class=""><br class="">
</div>
<div class="">No <font class="" face="Courier">.machines</font>
file has been created in the case folder. </div>
<div class=""> How can I successfully submit serial/parallel PBS
jobs? Thanks in advance for your help.</div>
<div class=""><br class="">
</div>
<div class="">Yoji Kobayashi</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: normal; letter-spacing: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px;">==========================================================<br
class="">
Yoji Kobayashi, Junior Assoc. Prof. <a
href="mailto:yojik@scl.kyoto-u.ac.jp" class=""
moz-do-not-send="true">yojik@scl.kyoto-u.ac.jp</a><br
class="">
<a href="http://www.scl.kyoto-u.ac.jp/%7Eyojik/index.htm"
class="" moz-do-not-send="true">http://www.scl.kyoto-u.ac.jp/~yojik/index.htm</a><br
class="">
<br class="">
Kageyama Group, Dept. of Energy and Hydrocarbon Chemistry<br
class="">
Graduate School of Engineering, Kyoto University<br
class="">
Nishikyo-ku, Kyoto 615-8510, Japan<br class="">
<br class="">
Tel.: +81-75-383-2509 Fax: +81-75-383-2510<br class="">
<a class="moz-txt-link-freetext" href="http://www.ehcc.kyoto-u.ac.jp/eh10/kageyama.html">http://www.ehcc.kyoto-u.ac.jp/eh10/kageyama.html</a><br
class="">
==========================================================</div>
</div>
</div>
</div>
</blockquote>
</body>
</html>