<div dir="ltr"><p class="MsoNormal" style="line-height:12.65pt;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;margin:0cm 0cm 10pt;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;font-family:"Times New Roman",serif">Dear Prof.
Peter Blaha and WIEN2k users,</span></p>
<p class="MsoNormal" style="line-height:12.65pt;margin:0cm 0cm 10pt;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;font-family:"Times New Roman",serif">Now I have loaded the openmpi/4.1.0
and compiled Wine2k. The admin told me that I can use your script in ></span><a href="http://www.wien2k.at/reg_user/faq/slurm.job" target="_blank" style="color:blue"><span style="font-size:12pt;font-family:"Times New Roman",serif">http://www.wien2k.at/reg_user/faq/slurm.job</span></a><span style="font-size:12pt;font-family:"Times New Roman",serif"> . I added
this lines to it too:</span></p>
<p class="MsoNormal" style="line-height:12.65pt;margin:0cm 0cm 10pt;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;font-family:"Times New Roman",serif;color:rgb(192,80,77)">module load openmpi/4.1.0_gcc620</span></p>
<p class="MsoNormal" style="line-height:12.65pt;margin:0cm 0cm 10pt;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;font-family:"Times New Roman",serif;color:rgb(192,80,77)">module load ifort</span></p>
<p class="MsoNormal" style="line-height:12.65pt;margin:0cm 0cm 10pt;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;font-family:"Times New Roman",serif;color:rgb(192,80,77)">module load mkl</span></p>
<p class="MsoNormal" style="line-height:12.65pt;margin:0cm 0cm 10pt;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;font-family:"Times New Roman",serif">but this error happened “<span style="color:rgb(192,80,77)">bash: mpirun: command not found</span>”.</span></p>
<p class="MsoNormal" style="line-height:12.65pt;margin:0cm 0cm 10pt;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:12pt;font-family:"Times New Roman",serif">In an interactive mode “<span style="color:rgb(31,73,125)">x lapw0 –p”</span> and “<span style="color:rgb(31,73,125)">x lapw2 –p</span>” are executed MPI but
“<span style="color:rgb(31,73,125)">x lapw1 –p</span>” is stoped
with following error:</span></p>
<p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="color:rgb(192,80,77)">w2k_dispatch_signal():
received: Segmentation fault</span></p>
<p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="color:rgb(192,80,77)">--------------------------------------------------------------------------</span></p>
<p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-family:"Times New Roman",serif">I noticed that the FFTW3 and OpenMPI installed on the cluster are
both compiled by gfortan. But I have compiled WIEN2k by intel ifort. I am not
sure whether the problem originates from this inconsistency between gfortan and
ifort.</span></p>
<p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-family:"Times New Roman",serif">I have checked that lapw1 has compiled correctly.</span></p>
<pre style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;margin:0cm 0cm 0.0001pt;font-size:10pt;font-family:Consolas"><span style="font-family:"Times New Roman",serif">Sincerely yours,</span></pre><pre style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;margin:0cm 0cm 0.0001pt;font-size:10pt;font-family:Consolas"><span style="font-family:"Times New Roman",serif">Leila</span></pre>
<p class="MsoNormal" style="margin:0cm 0cm 10pt;line-height:115%;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-family:"Times New Roman",serif"> </span></p></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 23, 2021 at 7:26 PM Peter Blaha <<a href="mailto:pblaha@theochem.tuwien.ac.at">pblaha@theochem.tuwien.ac.at</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Recompile with LI, since mpirun is supported (after loading the proper mpi).<br>
<br>
PS: Ask them if -np and -machinefile is still possible to use. Otherwise <br>
you cannot mix k-parallel and mpi parallel and for sure, for smaller <br>
cases it is a severe limitation to have only ONE mpi job with many <br>
k-points, small matrix size and many mpi cores.<br>
<br>
Am 23.04.2021 um 16:04 schrieb leila mollabashi:<br>
> Dear Prof. Peter Blaha and WIEN2k users,<br>
> <br>
> Thank you for your assistances.<br>
> <br>
> Here it is the admin reply:<br>
> <br>
> * mpirun/mpiexec command is supported after loadin propper module ( I<br>
> suggest openmpi/4.1.0 with gcc 6.2.0 or icc )<br>
> * you have to describe needed resources (I suggest : --nodes and<br>
> --ntasks-per-node , please use "whole node" , so ntasks-pper-node=<br>
> 28 or 32 or 48 , depending of partition)<br>
> * Yes, our cluster have "tight integration with mpi" but the<br>
> other-way-arround : our MPI libraries are compiled with SLURM<br>
> support, so when you describe resources at the beginning of batch<br>
> script, you do not have to use "-np" and "-machinefile" options for<br>
> mpirun/mpiexec<br>
> <br>
> * this error message " btl_openib_component.c:1699:init_one_device" is<br>
> caused by "old" mpi library, so please recompile your application<br>
> (WIEN2k) using openmpi/4.1.0_icc19<br>
> <br>
> Now should I compile WIEN2k with SL or LI?<br>
> <br>
> Sincerely yours,<br>
> <br>
> Leila Mollabashi<br>
> <br>
> <br>
> On Wed, Apr 14, 2021 at 10:34 AM Peter Blaha <br>
> <<a href="mailto:pblaha@theochem.tuwien.ac.at" target="_blank">pblaha@theochem.tuwien.ac.at</a> <mailto:<a href="mailto:pblaha@theochem.tuwien.ac.at" target="_blank">pblaha@theochem.tuwien.ac.at</a>>> wrote:<br>
> <br>
> It cannot initialize an mpi job, because it is missing the interface<br>
> software.<br>
> <br>
> You need to ask the computing center / system administrators how one<br>
> executes a mpi job on this computer.<br>
> <br>
> It could be, that "mpirun" is not supported on this machine. You may<br>
> try<br>
> a wien2k installation with system "LS" in siteconfig. This will<br>
> configure the parallel environment/commands using "slurm" commands like<br>
> srun -K -N_nodes_ -n_NP_ ..., replacing mpirun.<br>
> We used it once on our hpc machine, since it was recommended by the<br>
> computing center people. However, it turned out that the standard<br>
> mpirun<br>
> installation was more stable because the "slurm controller" died too<br>
> often leading to many random crashes. Anyway, if your system has<br>
> what is<br>
> called "tight integration of mpi", it might be necessary.<br>
> <br>
> Am 13.04.2021 um 21:47 schrieb leila mollabashi:<br>
> > Dear Prof. Peter Blaha and WIEN2k users,<br>
> ><br>
> > Then by run x lapw1 –p:<br>
> ><br>
> > starting parallel lapw1 at Tue Apr 13 21:04:15 CEST 2021<br>
> ><br>
> > -> starting parallel LAPW1 jobs at Tue Apr 13 21:04:15 CEST 2021<br>
> ><br>
> > running LAPW1 in parallel mode (using .machines)<br>
> ><br>
> > 2 number_of_parallel_jobs<br>
> ><br>
> > [1] 14530<br>
> ><br>
> > [e0467:14538] mca_base_component_repository_open: unable to open<br>
> > mca_btl_uct: libucp.so.0: cannot open shared object file: No such<br>
> file<br>
> > or directory (ignored)<br>
> ><br>
> > WARNING: There was an error initializing an OpenFabrics device.<br>
> ><br>
> > Local host: e0467<br>
> ><br>
> > Local device: mlx4_0<br>
> ><br>
> > MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD<br>
> ><br>
> > with errorcode 0.<br>
> ><br>
> > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.<br>
> ><br>
> > You may or may not see output from other processes, depending on<br>
> ><br>
> > exactly when Open MPI kills them.<br>
> ><br>
> ><br>
> --------------------------------------------------------------------------<br>
> ><br>
> > [e0467:14567] 1 more process has sent help message<br>
> > help-mpi-btl-openib.txt / error in device init<br>
> ><br>
> > [e0467:14567] 1 more process has sent help message<br>
> > help-mpi-btl-openib.txt / error in device init<br>
> ><br>
> > [e0467:14567] Set MCA parameter "orte_base_help_aggregate" to 0<br>
> to see<br>
> > all help / error messages<br>
> ><br>
> > [warn] Epoll MOD(1) on fd 27 failed. Old events were 6; read<br>
> change was<br>
> > 0 (none); write change was 2 (del): Bad file descriptor<br>
> ><br>
> >>Somewhere there should be some documentation how one runs an mpi<br>
> job on<br>
> > your system.<br>
> ><br>
> > Only I found this:<br>
> ><br>
> > Before ordering a task, it should be encapsulated in an appropriate<br>
> > script understandable for the queue system, e.g .:<br>
> ><br>
> > /home/users/user/<a href="http://submit_script.sl" rel="noreferrer" target="_blank">submit_script.sl</a> <<a href="http://submit_script.sl" rel="noreferrer" target="_blank">http://submit_script.sl</a>><br>
> <<a href="http://submit_script.sl" rel="noreferrer" target="_blank">http://submit_script.sl</a> <<a href="http://submit_script.sl" rel="noreferrer" target="_blank">http://submit_script.sl</a>>><br>
> ><br>
> > Sample SLURM script:<br>
> ><br>
> > #! / bin / bash -l<br>
> ><br>
> > #SBATCH -N 1<br>
> ><br>
> > #SBATCH --mem 5000<br>
> ><br>
> > #SBATCH --time = 20:00:00<br>
> ><br>
> > /sciezka/do/pliku/binarnego/<a href="http://plik_binarny.in" rel="noreferrer" target="_blank">plik_binarny.in</a><br>
> <<a href="http://plik_binarny.in" rel="noreferrer" target="_blank">http://plik_binarny.in</a>> <<a href="http://plik_binarny.in" rel="noreferrer" target="_blank">http://plik_binarny.in</a><br>
> <<a href="http://plik_binarny.in" rel="noreferrer" target="_blank">http://plik_binarny.in</a>>>><br>
> > /sciezka/do/pliku/wyjsciowego.out<br>
> ><br>
> > To order a task to a specific queue, use the #SBATCH -p<br>
> parameter, e.g.<br>
> ><br>
> > #! / bin / bash -l<br>
> ><br>
> > #SBATCH -N 1<br>
> ><br>
> > #SBATCH --mem 5000<br>
> ><br>
> > #SBATCH --time = 20:00:00<br>
> ><br>
> > #SBATCH -p standard<br>
> ><br>
> > /sciezka/do/pliku/binarnego/<a href="http://plik_binarny.in" rel="noreferrer" target="_blank">plik_binarny.in</a><br>
> <<a href="http://plik_binarny.in" rel="noreferrer" target="_blank">http://plik_binarny.in</a>> <<a href="http://plik_binarny.in" rel="noreferrer" target="_blank">http://plik_binarny.in</a><br>
> <<a href="http://plik_binarny.in" rel="noreferrer" target="_blank">http://plik_binarny.in</a>>>><br>
> > /siezka/do/pliku/wyjsciowego.out<br>
> ><br>
> > The task must then be ordered using the *sbatch* command<br>
> ><br>
> > sbatch /home/users/user/<a href="http://submit_script.sl" rel="noreferrer" target="_blank">submit_script.sl</a><br>
> <<a href="http://submit_script.sl" rel="noreferrer" target="_blank">http://submit_script.sl</a>> <<a href="http://submit_script.sl" rel="noreferrer" target="_blank">http://submit_script.sl</a><br>
> <<a href="http://submit_script.sl" rel="noreferrer" target="_blank">http://submit_script.sl</a>>><br>
> ><br>
> > *Ordering interactive tasks***<br>
> ><br>
> ><br>
> > Interactive tasks can be divided into two groups:<br>
> ><br>
> > ·interactive task (working in text mode)<br>
> ><br>
> > ·interactive task<br>
> ><br>
> > *Interactive task (working in text mode)***<br>
> ><br>
> ><br>
> > Ordering interactive tasks is very simple and in the simplest<br>
> case it<br>
> > comes down to issuing the command below.<br>
> ><br>
> > srun --pty / bin / bash<br>
> ><br>
> > Sincerely yours,<br>
> ><br>
> > Leila Mollabashi<br>
> ><br>
> ><br>
> > On Wed, Apr 14, 2021 at 12:03 AM leila mollabashi<br>
> > <<a href="mailto:le.mollabashi@gmail.com" target="_blank">le.mollabashi@gmail.com</a> <mailto:<a href="mailto:le.mollabashi@gmail.com" target="_blank">le.mollabashi@gmail.com</a>><br>
> <mailto:<a href="mailto:le.mollabashi@gmail.com" target="_blank">le.mollabashi@gmail.com</a> <mailto:<a href="mailto:le.mollabashi@gmail.com" target="_blank">le.mollabashi@gmail.com</a>>>><br>
> wrote:<br>
> ><br>
> > Dear Prof. Peter Blaha and WIEN2k users,<br>
> ><br>
> > Thank you for your assistances.<br>
> ><br>
> > > At least now the error: "lapw0 not found" is gone. Do you<br>
> > understand why ??<br>
> ><br>
> > Yes, I think that because now the path is clearly known.<br>
> ><br>
> > >How many slots do you get by this srun command ?<br>
> ><br>
> > Usually I went to node with 28 CPUs.<br>
> ><br>
> > >Is this the node with the name e0591 ???<br>
> ><br>
> > Yes, it is.<br>
> ><br>
> > >Of course the .machines file must be consistent (dynamically<br>
> adapted)<br>
> ><br>
> > with the actual nodename.<br>
> ><br>
> > Yes, to do this I use my script.<br>
> ><br>
> > >When I use “srun --pty -n 8 /bin/bash” that goes to the<br>
> node with 8 free<br>
> > cores, and run x lapw0 –p then this happens:<br>
> ><br>
> > starting parallel lapw0 at Tue Apr 13 20:50:49 CEST 2021<br>
> ><br>
> > -------- .machine0 : 4 processors<br>
> ><br>
> > [1] 12852<br>
> ><br>
> > [e0467:12859] mca_base_component_repository_open: unable to open<br>
> > mca_btl_uct: libucp.so.0: cannot open shared object file: No such<br>
> > file or directory (ignored)<br>
> ><br>
> > [e0467][[56319,1],1][btl_openib_component.c:1699:init_one_device]<br>
> > error obtaining device attributes for mlx4_0 errno says<br>
> Protocol not<br>
> > supported<br>
> ><br>
> > [e0467:12859] mca_base_component_repository_open: unable to open<br>
> > mca_pml_ucx: libucp.so.0: cannot open shared object file: No such<br>
> > file or directory (ignored)<br>
> ><br>
> > LAPW0 END<br>
> ><br>
> > [1] Done mpirun -np 4 -machinefile<br>
> > .machine0 /home/users/mollabashi/v19.2/lapw0_mpi lapw0.def >><br>
> .time00<br>
> ><br>
> > Sincerely yours,<br>
> ><br>
> > Leila Mollabashi<br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > Wien mailing list<br>
> > <a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a><br>
> <mailto:<a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a>><br>
> > <a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien" rel="noreferrer" target="_blank">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a><br>
> <<a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien" rel="noreferrer" target="_blank">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a>><br>
> > SEARCH the MAILING-LIST at:<br>
> <a href="http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html" rel="noreferrer" target="_blank">http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html</a><br>
> <<a href="http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html" rel="noreferrer" target="_blank">http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html</a>><br>
> ><br>
> <br>
> -- <br>
> --------------------------------------------------------------------------<br>
> Peter BLAHA, Inst.f. Materials Chemistry, TU Vienna, A-1060 Vienna<br>
> Phone: +43-1-58801-165300 FAX: +43-1-58801-165982<br>
> Email: <a href="mailto:blaha@theochem.tuwien.ac.at" target="_blank">blaha@theochem.tuwien.ac.at</a><br>
> <mailto:<a href="mailto:blaha@theochem.tuwien.ac.at" target="_blank">blaha@theochem.tuwien.ac.at</a>> WIEN2k: <a href="http://www.wien2k.at" rel="noreferrer" target="_blank">http://www.wien2k.at</a><br>
> <<a href="http://www.wien2k.at" rel="noreferrer" target="_blank">http://www.wien2k.at</a>><br>
> WWW: <a href="http://www.imc.tuwien.ac.at" rel="noreferrer" target="_blank">http://www.imc.tuwien.ac.at</a> <<a href="http://www.imc.tuwien.ac.at" rel="noreferrer" target="_blank">http://www.imc.tuwien.ac.at</a>><br>
> -------------------------------------------------------------------------<br>
> _______________________________________________<br>
> Wien mailing list<br>
> <a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a> <mailto:<a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a>><br>
> <a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien" rel="noreferrer" target="_blank">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a><br>
> <<a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien" rel="noreferrer" target="_blank">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a>><br>
> SEARCH the MAILING-LIST at:<br>
> <a href="http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html" rel="noreferrer" target="_blank">http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html</a><br>
> <<a href="http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html" rel="noreferrer" target="_blank">http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html</a>><br>
> <br>
> <br>
> _______________________________________________<br>
> Wien mailing list<br>
> <a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a><br>
> <a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien" rel="noreferrer" target="_blank">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a><br>
> SEARCH the MAILING-LIST at: <a href="http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html" rel="noreferrer" target="_blank">http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html</a><br>
> <br>
<br>
-- <br>
--------------------------------------------------------------------------<br>
Peter BLAHA, Inst.f. Materials Chemistry, TU Vienna, A-1060 Vienna<br>
Phone: +43-1-58801-165300 FAX: +43-1-58801-165982<br>
Email: <a href="mailto:blaha@theochem.tuwien.ac.at" target="_blank">blaha@theochem.tuwien.ac.at</a> WIEN2k: <a href="http://www.wien2k.at" rel="noreferrer" target="_blank">http://www.wien2k.at</a><br>
WWW: <a href="http://www.imc.tuwien.ac.at" rel="noreferrer" target="_blank">http://www.imc.tuwien.ac.at</a><br>
-------------------------------------------------------------------------<br>
_______________________________________________<br>
Wien mailing list<br>
<a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a><br>
<a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien" rel="noreferrer" target="_blank">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a><br>
SEARCH the MAILING-LIST at: <a href="http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html" rel="noreferrer" target="_blank">http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html</a><br>
</blockquote></div>