<div dir="ltr">Dear Pavel,<div><br></div><div>Many thanks again for your patience and guidance. With the <span style="color:rgb(0,0,0);font-family:monospace">libopenblas-openmp-dev</span> package it seems to work well!</div><div><br></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">$ ldd lapw1
</span><br> linux-vdso.so.1 (0x00007ffca83d8000)
<br> <b>libopenblas.so.0 => /lib/x86_64-linux-gnu/libopenblas.so.0 (0x000014563f924000)
</b><br> libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5 (0x000014563f65c000)
<br> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000014563f50d000)
<br> libmvec.so.1 => /lib/x86_64-linux-gnu/libmvec.so.1 (0x000014563f4e1000)
<br> libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1 (0x000014563f49f000)
<br> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x000014563f47c000)
<br> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000014563f288000)
<br> /lib64/ld-linux-x86-64.so.2 (0x0000145641b2d000)
<br> libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0 (0x000014563f23e000)
<br> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x000014563f223000)
<br> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x000014563f21d000)<br></span></div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><span style="background-color:rgb(255,255,255)"><font color="#444444"><br></font></span></div><div><span style="background-color:rgb(255,255,255)"><font color="#444444">and these options in siteconfig:</font></span></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)"> L Linker Flags: $(FOPT) -L/usr/lib/x86_64-linux-gnu/openblas-openmp
</span><br> R R_LIBS (LAPACK+BLAS): -lopenblas<br>
<br></span></div><div><font face="arial, sans-serif">I also get better timings now (though the TIME HAMILT are slightly longer, but overall improvement):</font></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">$ ../x lapw1
</span><br>STOP LAPW1 END
<br>119.400u 1.937s 0:32.53 372.9% 0+0k 0+37864io 0pf+0w<br>
</span><span style="font-family:monospace"><span style="color:rgb(0,0,0)">$ grep HORB *output1*
</span><br><span style="color:rgb(178,24,178)">test_case.output1</span><span style="color:rgb(24,178,178)">:</span><span style="color:rgb(0,0,0)"> TIME HAMILT (CPU) = 17.3, HNS = 18.4, </span><span style="font-weight:bold;color:rgb(255,84,84)">HORB</span><span style="color:rgb(0,0,0)"> = 0.0, DIAG = 85.0, SYNC = 0.0
</span><br><span style="color:rgb(178,24,178)">test_case.output1</span><span style="color:rgb(24,178,178)">:</span><span style="color:rgb(0,0,0)"> TIME HAMILT (WALL) = 4.6, HNS = 5.2, </span><span style="font-weight:bold;color:rgb(255,84,84)">HORB</span><span style="color:rgb(0,0,0)"> = 0.0, DIAG = 22.0, SYNC = 0.0</span><br></span></div><span style="background-color:rgb(255,255,255)"><font color="#444444"><div><span style="background-color:rgb(255,255,255)"><br></span></div><div>Thanks for your help,</div><div>David</div>---<br><b>Dr David Holec</b></font></span><div><i style="background-color:rgb(255,255,255)"><font color="#444444">Computational Materials Science group</font></i></div><div><span style="background-color:rgb(255,255,255)"><font color="#444444">Department of Materials Science<br>Montanuniversität Leoben<i><br></i></font></span></div><div><span style="background-color:rgb(255,255,255)"><font color="#444444"><br></font></span></div><div><i style="background-color:rgb(255,255,255)"><font color="#444444"><img src="https://docs.google.com/uc?export=download&id=1ckotMwDdxdBh15n7INYd0vppBJC4m3WF&revid=0BzAPFsUi1lR7b01FZnhQTWpZMUpuMVIvVE5mV3Zkdno3amZnPQ" width="64" height="96"><br></font></i></div><div><font color="#444444"><span style="background-color:rgb(255,255,255)"></span></font></div><div><span style="background-color:rgb(255,255,255)"><font color="#444444"><br>Franz-Josef-Strasse 18, A-8700 Leoben, Austria<br>tel. +43-(0)3842-4024211<br>fax. +43-(0)3842-4024202<br><a href="http://materials.unileoben.ac.at/" target="_blank">materials.unileoben.ac.at</a><br><a href="http://cms.unileoben.ac.at/" target="_blank">cms.unileoben.ac.at</a></font></span></div><div><span style="background-color:rgb(255,255,255)"><font color="#999999">________________________________</font></span></div><div><span style="background-color:rgb(255,255,255)"><font color="#999999">WHERE RESEARCH MEETS FUTURE</font></span></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 24 Nov 2021 at 12:39, Pavel Ondračka <<a href="mailto:pavel.ondracka@email.cz">pavel.ondracka@email.cz</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi David,<br>
<br>
well, it is hard to say without the debug info why the OpenBLAS crahes.<br>
My guess is that you link with the 64bit interface, try to install the<br>
standard one (openblas-openmp-devel) and replace openblas64-openmp with<br>
openblas-openmp everywhere in you config. Also remove the -lpthread<br>
(just to be safe, but in theory should not matter), it is not needed<br>
with OpenMP. If it still crashes, please recompile with debug info<br>
enabled (add -g to compiler options) and send me the x lapw1 output via<br>
PM.<br>
<br>
BTW my response was mostly motivated by me suspecting you actually link<br>
against slow netlib BLAS (which turned out to be the case) and I wanted<br>
to warn others in case someone in the future would be using your<br>
settings as a reference :-)<br>
<br>
Best regards<br>
Pavel<br>
<br>
On Wed, 2021-11-24 at 11:09 +0100, David Holec wrote:<br>
> Hi Pavel,<br>
> <br>
> Many thanks for your insights. As you know, I am not an expert on how<br>
> to compile codes, for me, this is sadly a trial and error adventure.<br>
> <br>
> I tried to compile it against the openblas library, but although the<br>
> compilation ends without any errors, I get a segmentation fault when<br>
> running lapw1 (on the test case<br>
> from <a href="http://www.wien2k.at/reg_user/benchmark/" rel="noreferrer" target="_blank">http://www.wien2k.at/reg_user/benchmark/</a>). The current setting<br>
> are:<br>
> <br>
> L Linker Flags: $(FOPT) -L/usr/lib/x86_64-linux-<br>
> gnu/openblas64-openmp<br>
> R R_LIBS (LAPACK+BLAS): /usr/lib/x86_64-linux-gnu/openblas64-<br>
> openmp/libopenblas64.so.0 -lpthread<br>
> <br>
> <br>
> (The rest is as I wrote in my first email.) Here is the list of<br>
> linked libraries:<br>
> $ ldd lapw1<br>
> linux-vdso.so.1 (0x00007ffea57d6000)<br>
> libopenblas64.so.0 => /lib/x86_64-linux-<br>
> gnu/libopenblas64.so.0 (0x000014fe2b2e5000)<br>
> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0<br>
> (0x000014fe2b2c2000)<br>
> libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5<br>
> (0x000014fe2affa000)<br>
> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6<br>
> (0x000014fe2aeab000)<br>
> libmvec.so.1 => /lib/x86_64-linux-gnu/libmvec.so.1<br>
> (0x000014fe2ae7f000)<br>
> libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1<br>
> (0x000014fe2ae3d000)<br>
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6<br>
> (0x000014fe2ac49000)<br>
> /lib64/ld-linux-x86-64.so.2 (0x000014fe2d4d3000)<br>
> libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0<br>
> (0x000014fe2abff000)<br>
> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1<br>
> (0x000014fe2abe4000)<br>
> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2<br>
> (0x000014fe2abde000)<br>
> <br>
> And here is the stacking fault (it doesn't tell me anything):<br>
> $x lapw1<br>
> <br>
> Program received signal SIGSEGV: Segmentation fault - invalid memory<br>
> reference.<br>
> <br>
> Backtrace for this error:<br>
> #0 0x148fb48b8d21 in ???<br>
> #1 0x148fb48b7ef5 in ???<br>
> #2 0x148fb452d20f in ???<br>
> #3 0x148fb54495fb in ???<br>
> Segmentation fault<br>
> 0.949u 0.472s 0:00.84 167.8% 0+0k 14400+10992io 31pf+0w<br>
> <br>
> Any idea?<br>
> <br>
> With the settings I reported yesterday, everything works just fine<br>
> (though probably not very efficiently - but this is not a problem for<br>
> me as this binary is not a "production" binary on any HPC):<br>
> <br>
> $ ldd lapw1<br>
> linux-vdso.so.1 (0x00007ffc60765000)<br>
> libblas.so.3 => /lib/x86_64-linux-gnu/libblas.so.3<br>
> (0x0000150cec90f000)<br>
> liblapack.so.3 => /lib/x86_64-linux-gnu/liblapack.so.3<br>
> (0x0000150cec26b000)<br>
> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0<br>
> (0x0000150cec248000)<br>
> libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5<br>
> (0x0000150cebf80000)<br>
> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6<br>
> (0x0000150cebe31000)<br>
> libmvec.so.1 => /lib/x86_64-linux-gnu/libmvec.so.1<br>
> (0x0000150cebe05000)<br>
> libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1<br>
> (0x0000150cebdc1000)<br>
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6<br>
> (0x0000150cebbcf000)<br>
> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1<br>
> (0x0000150cebbb4000)<br>
> /lib64/ld-linux-x86-64.so.2 (0x0000150cec9fc000)<br>
> libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0<br>
> (0x0000150cebb6a000)<br>
> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2<br>
> (0x0000150cebb64000)<br>
> <br>
> and:<br>
> <br>
> $ x lapw1<br>
> STOP LAPW1 END<br>
> 162.045u 0.918s 2:31.25 107.7% 0+0k 8976+37856io 35pf+0w<br>
> $ grep HORB *output1* <br>
> test_case.output1: TIME HAMILT (CPU) = 16.3, HNS = 20.0,<br>
> HORB = 0.0, DIAG = 125.9, SYNC = 0.0<br>
> test_case.output1: TIME HAMILT (WALL) = 4.4, HNS = 20.0,<br>
> HORB = 0.0, DIAG = 126.0, SYNC = 0.0<br>
> <br>
> (I am using it on my 10-year old "type writer":<br>
> $lscpu<br>
> Architecture: x86_64<br>
> CPU op-mode(s): 32-bit, 64-bit<br>
> Byte Order: Little Endian<br>
> Vendor ID: GenuineIntel<br>
> CPU family: 6<br>
> Model: 26<br>
> Model name: Intel(R) Xeon(R) CPU W3550<br>
> @ 3.07GHz<br>
> )<br>
> <br>
> <br>
> ---<br>
> Dr David Holec<br>
> Computational Materials Science group<br>
> Department of Materials Science<br>
> Montanuniversität Leoben<br>
> <br>
> <br>
> <br>
> Franz-Josef-Strasse 18, A-8700 Leoben, Austria<br>
> tel. +43-(0)3842-4024211<br>
> fax. +43-(0)3842-4024202<br>
> <a href="http://materials.unileoben.ac.at" rel="noreferrer" target="_blank">materials.unileoben.ac.at</a><br>
> <a href="http://cms.unileoben.ac.at" rel="noreferrer" target="_blank">cms.unileoben.ac.at</a><br>
> ________________________________<br>
> WHERE RESEARCH MEETS FUTURE<br>
> <br>
> <br>
> On Wed, 24 Nov 2021 at 08:27, Pavel Ondračka<br>
> <<a href="mailto:pavel.ondracka@email.cz" target="_blank">pavel.ondracka@email.cz</a>> wrote:<br>
> > Hi David,<br>
> > <br>
> > as you said it works for you, so feel free to ignore, but I have<br>
> > some<br>
> > further tips if you are interested. Ubuntu switches between the<br>
> > different blas and lapack using the "alternatives", so its<br>
> > difficult<br>
> > to<br>
> > say if you actually link with the correct one.<br>
> > <br>
> > "ldd lapw1" in WIENROOT should show which one is actually linked,<br>
> > what<br>
> > you want to have is the openmp openblas<br>
> > /usr/lib/x86_64-linux-gnu/openblas-openmp/libblas.so<br>
> > /usr/lib/x86_64-linux-gnu/openblas-openmp/liblapack.so<br>
> > or alternatively<br>
> > /usr/lib/x86_64-linux-gnu/openblas-openmp/libopenblas.so<br>
> > It looks like you linked with the pthread one. This is not a<br>
> > problem<br>
> > when running at single thread but at higher thread number this<br>
> > might<br>
> > lead to oversubscription and slowdowns as the pthreaded openblas<br>
> > doesn't respect the OMP_NUM_THREADS set by Wien2k. So I would<br>
> > recommend<br>
> > to relink with the openmp OpenBLAS. BTW it is usually safer to link<br>
> > with OpenBLAS explicitly using the -lopenblas instead of the -<br>
> > llapack<br>
> > -<br>
> > lblas to be sure you don't accidentally link the netlib one<br>
> > (libopenblas is just the libblas and libblapack provided by<br>
> > OpenBLAS<br>
> > merged together).<br>
> > <br>
> > In general easy way how to check performance is to run the serial<br>
> > test_case from <a href="http://www.wien2k.at/reg_user/benchmark/" rel="noreferrer" target="_blank">http://www.wien2k.at/reg_user/benchmark/</a> On modern<br>
> > CPUs<br>
> > (at least avx2) the runtime should be around 15-25 seconds at<br>
> > single<br>
> > thread.<br>
> > <br>
> > I see total runtime of ~18seconds on Fedora 35 with gfortran 11.2.1<br>
> > OpenBLAS and AMD Ryzen 9 3900X 12-Core Processor.<br>
> > Also look for the following line in test_case.output1, this is what<br>
> > I<br>
> > have:<br>
> > TIME HAMILT (WALL) = 2.2, HNS = 1.7, HORB = 0.0, DIAG<br>
> > = <br>
> > <br>
> > 14.0, SYNC = 0.0<br>
> > The time in HAMILT mostly depends on you compiler and vectorizing<br>
> > settings, while the DIAG is 99% lapack/blas related, so this can<br>
> > help<br>
> > with the diagnostics if things are slow.<br>
> > <br>
> > You might also get extra speedup of the HAMILT part by adding "-<br>
> > DHAVE_LIBMVEC" to the Compiler options.<br>
> > <br>
> > Best regards<br>
> > Pavel<br>
> > <br>
> > On Tue, 2021-11-23 at 11:07 +0100, David Holec wrote:<br>
> > > Dear all,<br>
> > > <br>
> > > I have just spent some time making Wien2k run on my single<br>
> > > machine<br>
> > > running Ubuntu 20.04 with gfortran/gcc. Since I am not an expert,<br>
> > it<br>
> > > was a trial and error, but it seems that I found a working<br>
> > combination<br>
> > > (sadly, the default parameters didn't work for me). Maybe this<br>
> > > will<br>
> > > help someone. Here are the settings that did the job for me:<br>
> > > <br>
> > > M OpenMP switch: -fopenmp<br>
> > > O Compiler options: -ffree-form -O2 -ftree-vectorize -<br>
> > > march=native -ffree-line-length-none -ffpe-summary=none<br>
> > > L Linker Flags: $(FOPT) -L/usr/lib/x86_64-linux-<br>
> > > gnu<br>
> > > P Preprocessor flags '-DParallel'<br>
> > > R R_LIBS (LAPACK+BLAS): -lblas -llapack -lpthread<br>
> > > F FFTW options: -DFFTW3 -DFFTW_OMP -I/usr/include<br>
> > > FFTW-LIBS: -L/usr/lib/x86_64-linux-gnu -<br>
> > > lfftw3<br>
> > -<br>
> > > lfftw3_omp<br>
> > > <br>
> > > where the FFTW options were:<br>
> > > <br>
> > > R FFTWROOT: /usr/<br>
> > > V FFTW_VERSION: FFTW3<br>
> > > L FFTW_LIB: lib/x86_64-linux-gnu<br>
> > > N FFTW_LIBNAME: fftw3<br>
> > > <br>
> > > Compiler versions:<br>
> > > $ gcc --version<br>
> > > gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0<br>
> > > gfortran --version<br>
> > > GNU Fortran (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0<br>
> > > <br>
> > > And I used the generic lapack and openblas packages provides by<br>
> > Ubuntu<br>
> > > repos:<br>
> > > liblapack-dev/focal,now 3.9.0-1build1 amd64 [installed]<br>
> > > liblapack3/focal,now 3.9.0-1build1 amd64 [installed,automatic]<br>
> > > <br>
> > > liblapack64-3/focal,now 3.9.0-1build1 amd64 [installed,automatic]<br>
> > > liblapack64-dev/focal,now 3.9.0-1build1 amd64 [installed]<br>
> > > <br>
> > > libblas-dev/focal,now 3.9.0-1build1 amd64 [installed]<br>
> > > libblas3/focal,now 3.9.0-1build1 amd64 [installed,automatic]<br>
> > > libblas64-3/focal,now 3.9.0-1build1 amd64 [installed,automatic]<br>
> > > libblas64-dev/focal,now 3.9.0-1build1 amd64 [installed,automatic]<br>
> > > <br>
> > > libopenblas64-0/focal-updates,now 0.3.8+ds-1ubuntu0.20.04.1 amd64<br>
> > > [installed]<br>
> > > libopenblas64-0-openmp/focal-updates,now 0.3.8+ds-<br>
> > > 1ubuntu0.20.04.1<br>
> > > amd64 [installed]<br>
> > > libopenblas64-0-pthread/focal-updates,now 0.3.8+ds-<br>
> > > 1ubuntu0.20.04.1<br>
> > > amd64 [installed,automatic]<br>
> > > <br>
> > > (I am not totally sure if I need all the libraries above, but<br>
> > > certainly, with these, the compilation seems to work and I am<br>
> > > able<br>
> > to<br>
> > > run SCF cycles & Telnes calculations without errors :-)<br>
> > > <br>
> > > All the best,<br>
> > > David<br>
> > > ---<br>
> > > Dr David Holec<br>
> > > Computational Materials Science group<br>
> > > Department of Materials Science<br>
> > > Montanuniversität Leoben<br>
> > > <br>
> > > <br>
> > > <br>
> > > Franz-Josef-Strasse 18, A-8700 Leoben, Austria<br>
> > > tel. +43-(0)3842-4024211<br>
> > > fax. +43-(0)3842-4024202<br>
> > > <a href="http://materials.unileoben.ac.at" rel="noreferrer" target="_blank">materials.unileoben.ac.at</a><br>
> > > <a href="http://cms.unileoben.ac.at" rel="noreferrer" target="_blank">cms.unileoben.ac.at</a><br>
> > > ________________________________<br>
> > > WHERE RESEARCH MEETS FUTURE<br>
> > > _______________________________________________<br>
> > > Wien mailing list<br>
> > > <a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a><br>
> > > <a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien" rel="noreferrer" target="_blank">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a><br>
> > > SEARCH the MAILING-LIST at: <br>
> > > <br>
> > <a href="http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html" rel="noreferrer" target="_blank">http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html</a><br>
> > <br>
> > _______________________________________________<br>
> > Wien mailing list<br>
> > <a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a><br>
> > <a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien" rel="noreferrer" target="_blank">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a><br>
> > SEARCH the MAILING-LIST at: <br>
> > <a href="http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html" rel="noreferrer" target="_blank">http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html</a><br>
> _______________________________________________<br>
> Wien mailing list<br>
> <a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a><br>
> <a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien" rel="noreferrer" target="_blank">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a><br>
> SEARCH the MAILING-LIST at: <br>
> <a href="http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html" rel="noreferrer" target="_blank">http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html</a><br>
<br>
_______________________________________________<br>
Wien mailing list<br>
<a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a><br>
<a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien" rel="noreferrer" target="_blank">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a><br>
SEARCH the MAILING-LIST at: <a href="http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html" rel="noreferrer" target="_blank">http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html</a><br>
</blockquote></div>