<div dir="auto"><p style="margin:0cm 0cm 10pt;line-height:18.0473px;font-size:11pt;font-family:calibri,sans-serif"><span style="font-size:12pt;line-height:19.688px;color:rgb(31,73,125)">Almost certainly the issue is in the error messages. For the remote casd:</span></p><p style="margin:0cm 0cm 10pt;line-height:18.0473px;font-size:11pt;font-family:calibri,sans-serif"><span style="font-size:12pt;line-height:19.688px;color:rgb(31,73,125)">"bash: lapw0: command not found"</span></p><p style="margin:0cm 0cm 10pt;line-height:18.0473px;font-size:11pt;font-family:calibri,sans-serif"><span style="font-size:12pt;line-height:19.688px;color:rgb(31,73,125)">Presumably you have not exported WIENROOT when you started your job, and/or it is not exported by openmpi. Check how to use mpi on your system including exporting PATH.</span></p><p style="margin:0cm 0cm 10pt;line-height:18.0473px;font-size:11pt;font-family:calibri,sans-serif">___</p><p style="margin:0cm 0cm 10pt;line-height:18.0473px;font-size:11pt;font-family:calibri,sans-serif">For the manual case:</p><p style="margin:0cm 0cm 10pt;line-height:18.0473px;font-family:calibri,sans-serif"><span style="font-size:12pt;line-height:19.688px;color:rgb(31,73,125)">"There are not enough slots available in the system to satisfy the 4 slots</span><span style="font-size:14.6667px"> that</span><span style="font-size:12pt;color:rgb(31,73,125)"> were requested by the application:"</span></p><p style="margin:0cm 0cm 10pt;line-height:18.0473px;font-family:calibri,sans-serif"><span style="font-size:12pt;color:rgb(31,73,125)">Since this is system specific you will need to look at what is needed locally to allocate cores for an interactive job.</span></p><br><div data-smartmail="gmail_signature" dir="auto">_____<br>Professor Laurence Marks<br>"Research is to see what everybody else has seen, and to think what nobody else has thought", Albert Szent-Györgyi<br><a href="http://www.numis.northwestern.edu">www.numis.northwestern.edu</a></div><br><div class="gmail_quote" dir="auto"><div dir="ltr" class="gmail_attr">On Sun, Mar 28, 2021, 15:38 leila mollabashi <<a href="mailto:le.mollabashi@gmail.com">le.mollabashi@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%">Dear Wien2k
users,</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%">I have a
problem with MPI parallelization while I have compiled the code with no error. The
v19.2 version of WIEN2k has been compiled with ifort, cc and openmpi. The mkl and
FFTW libraries were also used. On the SLURM queue cluster I can run with the
k-point parallel mode. But I could not run it mpi parallel mode even on 1 node. I used this script to run:</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">Sbatch <a href="https://urldefense.com/v3/__http://submit_script.sl__;!!Dq0X2DkFhyF93HkjWTBQKhk!GBjaGLxhtn9Pc7_neae0XGGQKa19d7Zchpp-25dfdrC1HpsVqnVQ7-WFBCfeOwPEeJq5iA$" target="_blank" rel="noreferrer">submit_script.sl</a></span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%">Which
<a href="https://urldefense.com/v3/__http://submit_script.sl__;!!Dq0X2DkFhyF93HkjWTBQKhk!GBjaGLxhtn9Pc7_neae0XGGQKa19d7Zchpp-25dfdrC1HpsVqnVQ7-WFBCfeOwPEeJq5iA$" target="_blank" rel="noreferrer">submit_script.sl</a> file is for example as follows:</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">#! /bin/bash -l</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">hostname</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">rm -fr .machines</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)"># for 4 cpus and kpoints (in input file)</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">nproc=4</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">#write .machines file</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">echo '#' .machines</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)"># example for an MPI parallel lapw0</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">echo 'lapw0:'`hostname`'</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">#:'$nproc >> .machines</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)"># k-point and mpi parallel lapw1/2</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">echo '1:'`hostname`':2' >> .machines</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">echo '1:'`hostname`':2' >> .machines</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">echo 'granularity:1' >>.machines</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">echo 'extrafine:1' >>.machines</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)"> run_lapw –p</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%">Then this
error appears:</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">error: command
/home/users/mollabashi/v19.2/lapw0para lapw0.def failed</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%">slurm-17032361.out
file is as follows:</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)"># .machines</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">bash: lapw0: command not found</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">real 0m0.001s</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">user 0m0.000s</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">sys 0m0.001s</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">grep: *scf1*: No such file or directory</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">grep: lapw2*.error: No such file or directory</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">> stop error</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%">Then when I
manually run this error appears:</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">There are not enough slots available in the system to
satisfy the 4</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">slots that were requested by the application:</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">
/home/users/mollabashi/v19.2/lapw0_mpi</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">Either request fewer slots for your application, or make
more slots</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">available for use.</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">A "slot" is the Open MPI term for an
allocatable unit where we can</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">launch a process.
The number of slots available are defined by the</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">environment in which Open MPI processes are run:</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)"> 1. Hostfile, via
"slots=N" clauses (N defaults to number of</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)"> processor
cores if not provided)</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)"> 2. The --host
command line parameter, via a ":N" suffix on the</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)"> hostname (N
defaults to 1 if not provided)</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)"> 3. Resource
manager (e.g., SLURM, PBS/Torque, LSF, etc.)</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)"> 4. If none of a
hostfile, the --host command line parameter, or an</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)"> RM is present,
Open MPI defaults to the number of processor cores</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">In all the above cases, if you want Open MPI to default
to the number</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">of hardware threads instead of the number of processor
cores, use the</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">--use-hwthread-cpus option.</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">Alternatively, you can use the --oversubscribe option to
ignore the</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">number of available slots when deciding the number of
processes to</span></p><p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;line-height:115%;color:rgb(31,73,125)">launch.</span></p></div>
</blockquote></div></div>