<div dir="ltr">Hi Pavel,<div><br></div><div>Many thanks for your insights. As you know, I am not an expert on how to compile codes, for me, this is sadly a trial and error adventure.</div><div><br></div><div>I tried to compile it against the openblas library, but although the compilation ends without any errors, I get a segmentation fault when running lapw1 (on the test case from <a href="http://www.wien2k.at/reg_user/benchmark/">http://www.wien2k.at/reg_user/benchmark/</a>). The current setting are:</div><div><br></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)"> L Linker Flags: $(FOPT) -L/usr/lib/x86_64-linux-gnu/openblas64-openmp
</span><br> R R_LIBS (LAPACK+BLAS): /usr/lib/x86_64-linux-gnu/openblas64-openmp/libopenblas64.so.0 -lpthread<br>
<br></span><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><span style="background-color:rgb(255,255,255)"><font color="#444444">(The rest is as I wrote in my first email.) Here is the list of linked libraries:</font></span></div><span style="background-color:rgb(255,255,255)"><font color="#444444"><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">$ ldd lapw1
</span><br> linux-vdso.so.1 (0x00007ffea57d6000)
<br> <b>libopenblas64.so.0 => /lib/x86_64-linux-gnu/libopenblas64.so.0 (0x000014fe2b2e5000)
</b><br> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x000014fe2b2c2000)
<br> libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5 (0x000014fe2affa000)
<br> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000014fe2aeab000)
<br> libmvec.so.1 => /lib/x86_64-linux-gnu/libmvec.so.1 (0x000014fe2ae7f000)
<br> libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1 (0x000014fe2ae3d000)
<br> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000014fe2ac49000)
<br> /lib64/ld-linux-x86-64.so.2 (0x000014fe2d4d3000)
<br> libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0 (0x000014fe2abff000)
<br> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x000014fe2abe4000)
<br> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x000014fe2abde000)<br>
<br></span></div><div><font face="arial, sans-serif">And here is the stacking fault (it doesn't tell me anything):</font></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">$x lapw1
</span><br>
<br>Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
<br>
<br>Backtrace for this error:
<br>#0 0x148fb48b8d21 in ???
<br>#1 0x148fb48b7ef5 in ???
<br>#2 0x148fb452d20f in ???
<br>#3 0x148fb54495fb in ???
<br>Segmentation fault
<br>0.949u 0.472s 0:00.84 167.8% 0+0k 14400+10992io 31pf+0w</span></div><div><span style="font-family:monospace"><br></span></div><div><font face="arial, sans-serif">Any idea?</font></div><div><span style="font-family:monospace"><br></span></div><div><font face="arial, sans-serif">With the settings I reported yesterday, everything works just fine (though probably not very efficiently - but this is not a problem for me as this binary is not a "production" binary on any HPC):</font></div><div><font face="arial, sans-serif"><br></font></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">$ ldd lapw1
</span><br> linux-vdso.so.1 (0x00007ffc60765000)
<br> <b>libblas.so.3 => /lib/x86_64-linux-gnu/libblas.so.3 (0x0000150cec90f000)
<br> liblapack.so.3 => /lib/x86_64-linux-gnu/liblapack.so.3 (0x0000150cec26b000)
</b><br> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x0000150cec248000)
<br> libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5 (0x0000150cebf80000)
<br> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x0000150cebe31000)
<br> libmvec.so.1 => /lib/x86_64-linux-gnu/libmvec.so.1 (0x0000150cebe05000)
<br> libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1 (0x0000150cebdc1000)
<br> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x0000150cebbcf000)
<br> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x0000150cebbb4000)
<br> /lib64/ld-linux-x86-64.so.2 (0x0000150cec9fc000)
<br> libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0 (0x0000150cebb6a000)
<br> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x0000150cebb64000)<br></span>
<br></div><div>and:</div><div><br></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">$ x lapw1
</span><br>STOP LAPW1 END
<br>162.045u 0.918s 2:31.25 107.7% 0+0k 8976+37856io 35pf+0w
<br><span style="color:rgb(0,0,0)">$ grep HORB *output1* </span><br><span style="color:rgb(178,24,178)">test_case.output1</span><span style="color:rgb(24,178,178)">:</span><span style="color:rgb(0,0,0)"> TIME HAMILT (CPU) = 16.3, HNS = 20.0, </span><span style="font-weight:bold;color:rgb(255,84,84)">HORB</span><span style="color:rgb(0,0,0)"> = 0.0, DIAG = 125.9, SYNC = 0.0
</span><br><span style="color:rgb(178,24,178)">test_case.output1</span><span style="color:rgb(24,178,178)">:</span><span style="color:rgb(0,0,0)"> TIME HAMILT (WALL) = 4.4, HNS = 20.0, </span><span style="font-weight:bold;color:rgb(255,84,84)">HORB</span><span style="color:rgb(0,0,0)"> = 0.0, DIAG = 126.0, SYNC = 0.0</span><br></span></div><div><br></div><div>(I am using it on my 10-year old "type writer":</div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">$lscpu
</span><br>Architecture: x86_64
<br>CPU op-mode(s): 32-bit, 64-bit
<br>Byte Order: Little Endian
<br>Vendor ID: GenuineIntel
<br>CPU family: 6
<br>Model: 26
<br>Model name: Intel(R) Xeon(R) CPU W3550 @ 3.07GHz</span></div><div><span style="font-family:monospace">)</span><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace"><br></span></div>---<br><b>Dr David Holec</b></font></span><div><i style="background-color:rgb(255,255,255)"><font color="#444444">Computational Materials Science group</font></i></div><div><span style="background-color:rgb(255,255,255)"><font color="#444444">Department of Materials Science<br>Montanuniversität Leoben<i><br></i></font></span></div><div><span style="background-color:rgb(255,255,255)"><font color="#444444"><br></font></span></div><div><i style="background-color:rgb(255,255,255)"><font color="#444444"><img src="https://docs.google.com/uc?export=download&id=1ckotMwDdxdBh15n7INYd0vppBJC4m3WF&revid=0BzAPFsUi1lR7b01FZnhQTWpZMUpuMVIvVE5mV3Zkdno3amZnPQ" width="64" height="96"><br></font></i></div><div><font color="#444444"><span style="background-color:rgb(255,255,255)"></span></font></div><div><span style="background-color:rgb(255,255,255)"><font color="#444444"><br>Franz-Josef-Strasse 18, A-8700 Leoben, Austria<br>tel. +43-(0)3842-4024211<br>fax. +43-(0)3842-4024202<br><a href="http://materials.unileoben.ac.at/" target="_blank">materials.unileoben.ac.at</a><br><a href="http://cms.unileoben.ac.at/" target="_blank">cms.unileoben.ac.at</a></font></span></div><div><span style="background-color:rgb(255,255,255)"><font color="#999999">________________________________</font></span></div><div><span style="background-color:rgb(255,255,255)"><font color="#999999">WHERE RESEARCH MEETS FUTURE</font></span></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 24 Nov 2021 at 08:27, Pavel Ondračka <<a href="mailto:pavel.ondracka@email.cz">pavel.ondracka@email.cz</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi David,<br>
<br>
as you said it works for you, so feel free to ignore, but I have some<br>
further tips if you are interested. Ubuntu switches between the<br>
different blas and lapack using the "alternatives", so its difficult to<br>
say if you actually link with the correct one.<br>
<br>
"ldd lapw1" in WIENROOT should show which one is actually linked, what<br>
you want to have is the openmp openblas<br>
/usr/lib/x86_64-linux-gnu/openblas-openmp/libblas.so<br>
/usr/lib/x86_64-linux-gnu/openblas-openmp/liblapack.so<br>
or alternatively<br>
/usr/lib/x86_64-linux-gnu/openblas-openmp/libopenblas.so<br>
It looks like you linked with the pthread one. This is not a problem<br>
when running at single thread but at higher thread number this might<br>
lead to oversubscription and slowdowns as the pthreaded openblas<br>
doesn't respect the OMP_NUM_THREADS set by Wien2k. So I would recommend<br>
to relink with the openmp OpenBLAS. BTW it is usually safer to link<br>
with OpenBLAS explicitly using the -lopenblas instead of the -llapack -<br>
lblas to be sure you don't accidentally link the netlib one<br>
(libopenblas is just the libblas and libblapack provided by OpenBLAS<br>
merged together).<br>
<br>
In general easy way how to check performance is to run the serial<br>
test_case from <a href="http://www.wien2k.at/reg_user/benchmark/" rel="noreferrer" target="_blank">http://www.wien2k.at/reg_user/benchmark/</a> On modern CPUs<br>
(at least avx2) the runtime should be around 15-25 seconds at single<br>
thread.<br>
<br>
I see total runtime of ~18seconds on Fedora 35 with gfortran 11.2.1<br>
OpenBLAS and AMD Ryzen 9 3900X 12-Core Processor.<br>
Also look for the following line in test_case.output1, this is what I<br>
have:<br>
TIME HAMILT (WALL) = 2.2, HNS = 1.7, HORB = 0.0, DIAG = <br>
14.0, SYNC = 0.0<br>
The time in HAMILT mostly depends on you compiler and vectorizing<br>
settings, while the DIAG is 99% lapack/blas related, so this can help<br>
with the diagnostics if things are slow.<br>
<br>
You might also get extra speedup of the HAMILT part by adding "-<br>
DHAVE_LIBMVEC" to the Compiler options.<br>
<br>
Best regards<br>
Pavel<br>
<br>
On Tue, 2021-11-23 at 11:07 +0100, David Holec wrote:<br>
> Dear all,<br>
> <br>
> I have just spent some time making Wien2k run on my single machine<br>
> running Ubuntu 20.04 with gfortran/gcc. Since I am not an expert, it<br>
> was a trial and error, but it seems that I found a working combination<br>
> (sadly, the default parameters didn't work for me). Maybe this will<br>
> help someone. Here are the settings that did the job for me:<br>
> <br>
> M OpenMP switch: -fopenmp<br>
> O Compiler options: -ffree-form -O2 -ftree-vectorize -<br>
> march=native -ffree-line-length-none -ffpe-summary=none<br>
> L Linker Flags: $(FOPT) -L/usr/lib/x86_64-linux-gnu<br>
> P Preprocessor flags '-DParallel'<br>
> R R_LIBS (LAPACK+BLAS): -lblas -llapack -lpthread<br>
> F FFTW options: -DFFTW3 -DFFTW_OMP -I/usr/include<br>
> FFTW-LIBS: -L/usr/lib/x86_64-linux-gnu -lfftw3 -<br>
> lfftw3_omp<br>
> <br>
> where the FFTW options were:<br>
> <br>
> R FFTWROOT: /usr/<br>
> V FFTW_VERSION: FFTW3<br>
> L FFTW_LIB: lib/x86_64-linux-gnu<br>
> N FFTW_LIBNAME: fftw3<br>
> <br>
> Compiler versions:<br>
> $ gcc --version<br>
> gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0<br>
> gfortran --version<br>
> GNU Fortran (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0<br>
> <br>
> And I used the generic lapack and openblas packages provides by Ubuntu<br>
> repos:<br>
> liblapack-dev/focal,now 3.9.0-1build1 amd64 [installed]<br>
> liblapack3/focal,now 3.9.0-1build1 amd64 [installed,automatic]<br>
> <br>
> liblapack64-3/focal,now 3.9.0-1build1 amd64 [installed,automatic]<br>
> liblapack64-dev/focal,now 3.9.0-1build1 amd64 [installed]<br>
> <br>
> libblas-dev/focal,now 3.9.0-1build1 amd64 [installed]<br>
> libblas3/focal,now 3.9.0-1build1 amd64 [installed,automatic]<br>
> libblas64-3/focal,now 3.9.0-1build1 amd64 [installed,automatic]<br>
> libblas64-dev/focal,now 3.9.0-1build1 amd64 [installed,automatic]<br>
> <br>
> libopenblas64-0/focal-updates,now 0.3.8+ds-1ubuntu0.20.04.1 amd64<br>
> [installed]<br>
> libopenblas64-0-openmp/focal-updates,now 0.3.8+ds-1ubuntu0.20.04.1<br>
> amd64 [installed]<br>
> libopenblas64-0-pthread/focal-updates,now 0.3.8+ds-1ubuntu0.20.04.1<br>
> amd64 [installed,automatic]<br>
> <br>
> (I am not totally sure if I need all the libraries above, but<br>
> certainly, with these, the compilation seems to work and I am able to<br>
> run SCF cycles & Telnes calculations without errors :-)<br>
> <br>
> All the best,<br>
> David<br>
> ---<br>
> Dr David Holec<br>
> Computational Materials Science group<br>
> Department of Materials Science<br>
> Montanuniversität Leoben<br>
> <br>
> <br>
> <br>
> Franz-Josef-Strasse 18, A-8700 Leoben, Austria<br>
> tel. +43-(0)3842-4024211<br>
> fax. +43-(0)3842-4024202<br>
> <a href="http://materials.unileoben.ac.at" rel="noreferrer" target="_blank">materials.unileoben.ac.at</a><br>
> <a href="http://cms.unileoben.ac.at" rel="noreferrer" target="_blank">cms.unileoben.ac.at</a><br>
> ________________________________<br>
> WHERE RESEARCH MEETS FUTURE<br>
> _______________________________________________<br>
> Wien mailing list<br>
> <a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a><br>
> <a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien" rel="noreferrer" target="_blank">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a><br>
> SEARCH the MAILING-LIST at: <br>
> <a href="http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html" rel="noreferrer" target="_blank">http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html</a><br>
<br>
_______________________________________________<br>
Wien mailing list<br>
<a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a><br>
<a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien" rel="noreferrer" target="_blank">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a><br>
SEARCH the MAILING-LIST at: <a href="http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html" rel="noreferrer" target="_blank">http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html</a><br>
</blockquote></div>