<div dir="auto">I think we (collectively) may be confusing things by offering too much advice!<div dir="auto"><br></div><div dir="auto">Let's keep it simple, and focus on one thing at a time. The "mpirun not found" has nothing to do with compilers. It is 100% due to your not having the PATH variable set right. This is not fftw, but probably in the general options and mpi module. (It is not in the compiler.)</div><div dir="auto"><br></div><div dir="auto">The library not being found is similarly due to LD_LIBRARY_PATH not being right, or perhaps nfs mounting issues. This may be from the compiler variables (mpiifort for Intel), which may not be correctly set by the module.</div><div dir="auto"><br></div><div dir="auto">I suggest that you focus on the PATH first, using</div><div dir="auto">which mpirun</div><div dir="auto">which lapw1</div><div dir="auto">echo $PATH</div><div dir="auto">echo $WIENROOT</div><div dir="auto"><br></div><div dir="auto">When this is correct in the script that runs your job we can move forward and solve the library issue using</div><div dir="auto">echo $LD_LIBRARY_PATH</div><div dir="auto">ldd $WIENROOT/lapw1_mpi</div><div dir="auto">ldd $WIENROOT/lapw0_mpi</div><div dir="auto"><br></div><div dir="auto">Please focus just of the PATH first. If you have problems, find a way (DropBox/Drive etc) to post the script and result.<br><br><div data-smartmail="gmail_signature" dir="auto">_____<br>Professor Laurence Marks<br>"Research is to see what everybody else has seen, and to think what nobody else has thought", Albert Szent-Györgyi<br><a href="http://www.numis.northwestern.edu">www.numis.northwestern.edu</a></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, May 4, 2021, 22:24 Gavin Abo <<a href="mailto:gsabo@crimson.ua.edu">gsabo@crimson.ua.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<p><font color="#0000ff">Three additional comments:</font></p>
<p><font color="#0000ff">1) If you are running the slurm.job script
as Non-Interactive [1,2], you might need a "source
/etc/profile.d/ummodules.csh" line like that at [3].<br>
</font></p>
<font color="#0000ff">[1]
<a href="https://urldefense.com/v3/__https://slurm.schedmd.com/faq.html*sbatch_srun__;Iw!!Dq0X2DkFhyF93HkjWTBQKhk!C5RwGaeHepJtMl42xVZlEAUWtk2DM4zXGp3jfTPI5NJGAplXozUMwOc-7I4TfdoCMq2-Yw$" target="_blank" rel="noreferrer">https://slurm.schedmd.com/faq.html#sbatch_srun</a></font><br>
<font color="#0000ff">[2]
<a href="https://urldefense.com/v3/__https://wiki.umiacs.umd.edu/umiacs/index.php/SLURM/JobSubmission__;!!Dq0X2DkFhyF93HkjWTBQKhk!C5RwGaeHepJtMl42xVZlEAUWtk2DM4zXGp3jfTPI5NJGAplXozUMwOc-7I4TfdqwsBiRag$" target="_blank" rel="noreferrer">https://wiki.umiacs.umd.edu/umiacs/index.php/SLURM/JobSubmission</a></font><br>
<font color="#0000ff">[3]
<a href="https://urldefense.com/v3/__https://wiki.umiacs.umd.edu/umiacs/index.php/Modules__;!!Dq0X2DkFhyF93HkjWTBQKhk!C5RwGaeHepJtMl42xVZlEAUWtk2DM4zXGp3jfTPI5NJGAplXozUMwOc-7I4TfdrUX-bsYA$" target="_blank" rel="noreferrer">https://wiki.umiacs.umd.edu/umiacs/index.php/Modules</a></font><br>
<p><font color="#0000ff"> 2) To eliminate any possible conflicts
between Intel compilers (ifort, icc) and GNU compilers
(gfortran, gcc) such as those mentioned in posts [4,5]. I
suggest compiling of WIEN2k using the same C compiler and
Fortran compiler that Open MPI was compiled with.</font></p>
<p><font color="#0000ff">The commands [6] below might help you check
that the Linux environment is using the intended Open MPI and
mpi parallel compilers [7]:<br>
</font></p>
<p>username@computername:~/Desktop$ mpirun -V<br>
mpirun (Open MPI) 4.1.1<br>
<br>
Report bugs to <a href="https://urldefense.com/v3/__http://www.open-mpi.org/community/help/__;!!Dq0X2DkFhyF93HkjWTBQKhk!C5RwGaeHepJtMl42xVZlEAUWtk2DM4zXGp3jfTPI5NJGAplXozUMwOc-7I4TfdpyptJ4aw$" target="_blank" rel="noreferrer">http://www.open-mpi.org/community/help/</a><br>
username@computername:~/Desktop$ mpicc --showme:version<br>
mpicc: Open MPI 4.1.1 (Language: C)<br>
username@computername:~/Desktop$ mpifort --showme:version<br>
mpifort: Open MPI 4.1.1 (Language: Fortran)<br>
username@computername:~/Desktop$ mpicc --version<br>
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0<br>
Copyright (C) 2019 Free Software Foundation, Inc.<br>
This is free software; see the source for copying conditions.
There is NO<br>
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
PURPOSE.<br>
<br>
username@computername:~/Desktop$ mpifort --version<br>
GNU Fortran (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0<br>
Copyright (C) 2019 Free Software Foundation, Inc.<br>
This is free software; see the source for copying conditions.
There is NO<br>
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
PURPOSE.<br>
</p>
<font color="#0000ff">[4]
<a href="https://urldefense.com/v3/__https://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/msg08315.html__;!!Dq0X2DkFhyF93HkjWTBQKhk!C5RwGaeHepJtMl42xVZlEAUWtk2DM4zXGp3jfTPI5NJGAplXozUMwOc-7I4Tfdqey1lHyg$" target="_blank" rel="noreferrer">https://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/msg08315.html</a></font><br>
<font color="#0000ff">[5]
<a href="https://urldefense.com/v3/__https://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/msg17052.html__;!!Dq0X2DkFhyF93HkjWTBQKhk!C5RwGaeHepJtMl42xVZlEAUWtk2DM4zXGp3jfTPI5NJGAplXozUMwOc-7I4TfdraaFQRnw$" target="_blank" rel="noreferrer">https://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/msg17052.html</a></font><br>
<font color="#0000ff">[6]
<a href="https://urldefense.com/v3/__https://stackoverflow.com/questions/10056898/how-do-you-check-the-version-of-openmpi__;!!Dq0X2DkFhyF93HkjWTBQKhk!C5RwGaeHepJtMl42xVZlEAUWtk2DM4zXGp3jfTPI5NJGAplXozUMwOc-7I4Tfdph7KJdew$" target="_blank" rel="noreferrer">https://stackoverflow.com/questions/10056898/how-do-you-check-the-version-of-openmpi</a></font><br>
<font color="#0000ff">[7]
<a href="https://urldefense.com/v3/__https://www.open-mpi.org/faq/?category=mpi-apps*general-build__;Iw!!Dq0X2DkFhyF93HkjWTBQKhk!C5RwGaeHepJtMl42xVZlEAUWtk2DM4zXGp3jfTPI5NJGAplXozUMwOc-7I4TfdosQiVCzQ$" target="_blank" rel="noreferrer">https://www.open-mpi.org/faq/?category=mpi-apps#general-build</a><br>
</font>
<p><font color="#0000ff">3)</font> <font color="#0000ff">Those
cluster administrators are usually more savvy than I am with
installation and optimization of software (using compiler
documentation, e.g. [8,9]) on a high performance computing (hpc)
supercomputer [10,11]. They would know your situation better.
For example, they could login to their administrator account on
the cluster to install WIEN2k only in your user account
directory (/home/users/mollabashi), and they would know how to
set the appropriate access permissions [12]. Alternatively, if
your not using a personal laptop but a computer at the
organization to remotely connect to the cluster then they might
use remote desktop access </font><font color="#0000ff"><font color="#0000ff">[13] </font>to help you with the installation
within only your account. Or they might use another method.<br>
</font></p>
<font color="#0000ff">[8]
<a href="https://urldefense.com/v3/__https://software.intel.com/content/www/us/en/develop/articles/download-documentation-intel-compiler-current-and-previous.html__;!!Dq0X2DkFhyF93HkjWTBQKhk!C5RwGaeHepJtMl42xVZlEAUWtk2DM4zXGp3jfTPI5NJGAplXozUMwOc-7I4TfdqQX1Yugg$" target="_blank" rel="noreferrer">https://software.intel.com/content/www/us/en/develop/articles/download-documentation-intel-compiler-current-and-previous.html</a></font><br>
<font color="#0000ff">[9] <a href="https://urldefense.com/v3/__https://gcc.gnu.org/onlinedocs/__;!!Dq0X2DkFhyF93HkjWTBQKhk!C5RwGaeHepJtMl42xVZlEAUWtk2DM4zXGp3jfTPI5NJGAplXozUMwOc-7I4TfdobQb5a1g$" target="_blank" rel="noreferrer">https://gcc.gnu.org/onlinedocs/</a></font><br>
<font color="#0000ff">[10]
<a href="https://urldefense.com/v3/__https://www.usgs.gov/core-science-systems/sas/arc/about/what-high-performance-computing__;!!Dq0X2DkFhyF93HkjWTBQKhk!C5RwGaeHepJtMl42xVZlEAUWtk2DM4zXGp3jfTPI5NJGAplXozUMwOc-7I4TfdqNzk9j6g$" target="_blank" rel="noreferrer">https://www.usgs.gov/core-science-systems/sas/arc/about/what-high-performance-computing</a></font><br>
<font color="#0000ff">[11]
<a href="https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Supercomputer__;!!Dq0X2DkFhyF93HkjWTBQKhk!C5RwGaeHepJtMl42xVZlEAUWtk2DM4zXGp3jfTPI5NJGAplXozUMwOc-7I4Tfdr6pni8Lw$" target="_blank" rel="noreferrer">https://en.wikipedia.org/wiki/Supercomputer</a></font><br>
<font color="#0000ff">[12]
<a href="https://urldefense.com/v3/__https://www.oreilly.com/library/view/running-linux-third/156592469X/ch04s14.html__;!!Dq0X2DkFhyF93HkjWTBQKhk!C5RwGaeHepJtMl42xVZlEAUWtk2DM4zXGp3jfTPI5NJGAplXozUMwOc-7I4TfdroF-ZgDw$" target="_blank" rel="noreferrer">https://www.oreilly.com/library/view/running-linux-third/156592469X/ch04s14.html</a><br>
</font>
<div><font color="#0000ff">[13]
<a href="https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Desktop_sharing__;!!Dq0X2DkFhyF93HkjWTBQKhk!C5RwGaeHepJtMl42xVZlEAUWtk2DM4zXGp3jfTPI5NJGAplXozUMwOc-7I4Tfdr-07_1qQ$" target="_blank" rel="noreferrer">https://en.wikipedia.org/wiki/Desktop_sharing</a></font><br>
</div>
<div><br>
</div>
<div>On 5/4/2021 3:40 PM, Laurence Marks
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_default" style="font-family:verdana,sans-serif;color:#000000">
<div dir="ltr" style="color:rgb(34,34,34)">
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(0,0,0)">For
certain, "<span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)">/opt/exp_soft/local/generic/</span><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)">openmpi/4.1.0_gcc620/bin/</span><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)">mpiexec
/home/users/mollabashi/codes/</span><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)">v21.1/run_lapw
-p" is completely wrong. You do not, repear do not use
mpirun or mpiexec to start run_lapw. It has to be
started by simply "run_lapw -p ..." by itself.</span></div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(0,0,0)"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)"><br>
</span></div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(0,0,0)"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)">I
suggest that you create a very simple job which has
the commands:</span></div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(0,0,0)"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)"><br>
</span></div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(0,0,0)"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)">which
mpirun</span></div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(0,0,0)"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)">which
lapw1_mpi</span></div>
<div class="gmail_default" style="font-family:verdana,sans-serif">echo $WIENROOT</div>
<div class="gmail_default" style="font-family:verdana,sans-serif">ldd
$WIENROOT/lapw1_mpi</div>
<div class="gmail_default" style="font-family:verdana,sans-serif">ldd
$WIENROOT/lapw1</div>
<div class="gmail_default" style="font-family:verdana,sans-serif">echo env</div>
<div class="gmail_default" style="font-family:verdana,sans-serif">echo $PATH</div>
<div class="gmail_default" style="font-family:verdana,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif">Run this
interactively as well as in a batch job and compare. You
will find that there are something which are not present
when you are launching your slurm job that are present
interactively. You need to repair these with relevant
PATH/LD_LIBRARY_PATH etc</div>
<div class="gmail_default" style="font-family:verdana,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif">Your problems are
not Wien2k problems, they are due to incorrect
modules/script/environment or similar. Have you asked
your sysadmin for help? I am certain that someone local
who is experienced with standard linux can tell you very
quickly what to do. </div>
<div class="gmail_default" style="font-family:verdana,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif">N.B., there is an
error in your path setting.</div>
</div>
<div style="color:rgb(34,34,34)">
<div style="margin:5px 0px">
<div id="m_-1356530579119140gmail-q_126" style="font-size:11px;line-height:6px;background-color:rgb(232,234,237);border:none;clear:both;outline:none;width:24px;color:rgb(80,0,80);border-radius:5.5px"><br>
</div>
</div>
</div>
</div>
</div>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, May 4, 2021 at 3:38
PM leila mollabashi <<a href="mailto:le.mollabashi@gmail.com" target="_blank" rel="noreferrer">le.mollabashi@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">Dear all WIEN2k users,<br>
Thank you for your guides.
<div>>take care on the correct location ...<br>
It is the /usr/share/Modules/init<br>
After adding the “source /usr/share/Modules/init/tcsh”
line in to the script the same error appeared:<br>
mpirun: command not found</div>
<div><br>
In fact, with and without “source
/usr/share/Modules/init/tcsh” it is written in
slurm.out file that “
module load complete
”.</div>
<div><br>
</div>
<div>I noticed that “export” is also the bash command so
I used these commands to path the openmpi and fftw:<br>
setenv LD_LIBRARY_PATH
{$LD_LIBRARY_PATH}:/opt/exp_soft/local/generic/openmpi/4.1.0_gcc620/lib:/opt/exp_soft/local/generic/fftw/3.3.8_gcc620/lib<br>
set path = ($path
/opt/exp_soft/local/generic/openmpi/4.1.0_gcc620/bin:/opt/exp_soft/local/generic/fftw/3.3.8_gcc620/bin)</div>
<div>But result is the same:<br>
bash: mpirun: command not found<br>
</div>
<div><br>
By using this line in the script:<br>
/opt/exp_soft/local/generic/openmpi/4.1.0_gcc620/bin/mpiexec
/home/users/mollabashi/codes/v21.1/run_lapw -p<br>
The calculation stopped with the following error:<br>
mpirun does not support recursive calls</div>
<div><br>
> I wonder that you have only single modules…<br>
There are different versions of ifort and mkl:
ifort/15.0.0, ifort/15.0.3, ifort/17.0.1,
ifort/19.1.3.304(default) mkl/11.2, mkl/<a href="https://urldefense.com/v3/__http://11.2.3.187__;!!Dq0X2DkFhyF93HkjWTBQKhk!BD88XA2ujgG8Gel0NZaSOKaSrtPN7kq75O9hkG-dnIZQJRbsnQE-ArEqbFqA6XVu7qQFcg$" target="_blank" rel="noreferrer">11.2.3.187</a> mkl/2017.1.132,
mkl/2019.2.187, mkl/2020.0.4(default). I used the
defaults<br>
> you may also wish to make a single module file to
be loaded…<br>
That is a good idea.<br>
> On our cluster we have different W2k modules ….<br>
As you know WIEN2k is not a free code and the users of
the cluster that I am using are not registered WIEN2k
users. Thus, according to my moral commitment to the
WIEN2k developers, I cannot ask the administrator to
install it on the cluster. I should install it on my
user account.</div>
<div><br>
</div>
<div>Sincerely yours,<br>
Leila<br>
>PS.: maybe one should mention this tcsh "problem"
in the slurm.job example on the FAQ page by adding (or
similar)…<br>
That is a good idea. Thank you for your suggestion.<br>
</div>
</div>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr"><br>
</div>
</div>
</div>
</blockquote>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">Professor Laurence Marks<br>
Department of Materials Science and Engineering<br>
Northwestern University<br>
<a href="http://www.numis.northwestern.edu/" target="_blank" rel="noreferrer">www.numis.northwestern.edu</a>
<div>"Research is to see what everybody else has seen, and
to think what nobody else has thought" Albert <span style="font-family:Arial,Helvetica,sans-serif;font-size:12.8px">Szent-</span><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)">Györgyi</span></div>
</div>
</div>
</div>
</blockquote>
</div>
_______________________________________________<br>
Wien mailing list<br>
<a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank" rel="noreferrer">Wien@zeus.theochem.tuwien.ac.at</a><br>
<a href="https://urldefense.com/v3/__http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien__;!!Dq0X2DkFhyF93HkjWTBQKhk!C5RwGaeHepJtMl42xVZlEAUWtk2DM4zXGp3jfTPI5NJGAplXozUMwOc-7I4Tfdoe8NT97g$" rel="noreferrer noreferrer" target="_blank">https://urldefense.com/v3/__http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien__;!!Dq0X2DkFhyF93HkjWTBQKhk!C5RwGaeHepJtMl42xVZlEAUWtk2DM4zXGp3jfTPI5NJGAplXozUMwOc-7I4Tfdoe8NT97g$</a> <br>
SEARCH the MAILING-LIST at: <a href="https://urldefense.com/v3/__http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html__;!!Dq0X2DkFhyF93HkjWTBQKhk!C5RwGaeHepJtMl42xVZlEAUWtk2DM4zXGp3jfTPI5NJGAplXozUMwOc-7I4TfdrbLIpdvQ$" rel="noreferrer noreferrer" target="_blank">https://urldefense.com/v3/__http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html__;!!Dq0X2DkFhyF93HkjWTBQKhk!C5RwGaeHepJtMl42xVZlEAUWtk2DM4zXGp3jfTPI5NJGAplXozUMwOc-7I4TfdrbLIpdvQ$</a> <br>
</blockquote></div>