[Wien] Compiling Wien2k 21.1 on Ubuntu 20.04 with gfortran
Pavel Ondračka
pavel.ondracka at email.cz
Wed Nov 24 12:39:05 CET 2021
Hi David,
well, it is hard to say without the debug info why the OpenBLAS crahes.
My guess is that you link with the 64bit interface, try to install the
standard one (openblas-openmp-devel) and replace openblas64-openmp with
openblas-openmp everywhere in you config. Also remove the -lpthread
(just to be safe, but in theory should not matter), it is not needed
with OpenMP. If it still crashes, please recompile with debug info
enabled (add -g to compiler options) and send me the x lapw1 output via
PM.
BTW my response was mostly motivated by me suspecting you actually link
against slow netlib BLAS (which turned out to be the case) and I wanted
to warn others in case someone in the future would be using your
settings as a reference :-)
Best regards
Pavel
On Wed, 2021-11-24 at 11:09 +0100, David Holec wrote:
> Hi Pavel,
>
> Many thanks for your insights. As you know, I am not an expert on how
> to compile codes, for me, this is sadly a trial and error adventure.
>
> I tried to compile it against the openblas library, but although the
> compilation ends without any errors, I get a segmentation fault when
> running lapw1 (on the test case
> from http://www.wien2k.at/reg_user/benchmark/). The current setting
> are:
>
> L Linker Flags: $(FOPT) -L/usr/lib/x86_64-linux-
> gnu/openblas64-openmp
> R R_LIBS (LAPACK+BLAS): /usr/lib/x86_64-linux-gnu/openblas64-
> openmp/libopenblas64.so.0 -lpthread
>
>
> (The rest is as I wrote in my first email.) Here is the list of
> linked libraries:
> $ ldd lapw1
> linux-vdso.so.1 (0x00007ffea57d6000)
> libopenblas64.so.0 => /lib/x86_64-linux-
> gnu/libopenblas64.so.0 (0x000014fe2b2e5000)
> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> (0x000014fe2b2c2000)
> libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5
> (0x000014fe2affa000)
> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
> (0x000014fe2aeab000)
> libmvec.so.1 => /lib/x86_64-linux-gnu/libmvec.so.1
> (0x000014fe2ae7f000)
> libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1
> (0x000014fe2ae3d000)
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
> (0x000014fe2ac49000)
> /lib64/ld-linux-x86-64.so.2 (0x000014fe2d4d3000)
> libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0
> (0x000014fe2abff000)
> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
> (0x000014fe2abe4000)
> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
> (0x000014fe2abde000)
>
> And here is the stacking fault (it doesn't tell me anything):
> $x lapw1
>
> Program received signal SIGSEGV: Segmentation fault - invalid memory
> reference.
>
> Backtrace for this error:
> #0 0x148fb48b8d21 in ???
> #1 0x148fb48b7ef5 in ???
> #2 0x148fb452d20f in ???
> #3 0x148fb54495fb in ???
> Segmentation fault
> 0.949u 0.472s 0:00.84 167.8% 0+0k 14400+10992io 31pf+0w
>
> Any idea?
>
> With the settings I reported yesterday, everything works just fine
> (though probably not very efficiently - but this is not a problem for
> me as this binary is not a "production" binary on any HPC):
>
> $ ldd lapw1
> linux-vdso.so.1 (0x00007ffc60765000)
> libblas.so.3 => /lib/x86_64-linux-gnu/libblas.so.3
> (0x0000150cec90f000)
> liblapack.so.3 => /lib/x86_64-linux-gnu/liblapack.so.3
> (0x0000150cec26b000)
> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> (0x0000150cec248000)
> libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5
> (0x0000150cebf80000)
> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
> (0x0000150cebe31000)
> libmvec.so.1 => /lib/x86_64-linux-gnu/libmvec.so.1
> (0x0000150cebe05000)
> libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1
> (0x0000150cebdc1000)
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
> (0x0000150cebbcf000)
> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
> (0x0000150cebbb4000)
> /lib64/ld-linux-x86-64.so.2 (0x0000150cec9fc000)
> libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0
> (0x0000150cebb6a000)
> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
> (0x0000150cebb64000)
>
> and:
>
> $ x lapw1
> STOP LAPW1 END
> 162.045u 0.918s 2:31.25 107.7% 0+0k 8976+37856io 35pf+0w
> $ grep HORB *output1*
> test_case.output1: TIME HAMILT (CPU) = 16.3, HNS = 20.0,
> HORB = 0.0, DIAG = 125.9, SYNC = 0.0
> test_case.output1: TIME HAMILT (WALL) = 4.4, HNS = 20.0,
> HORB = 0.0, DIAG = 126.0, SYNC = 0.0
>
> (I am using it on my 10-year old "type writer":
> $lscpu
> Architecture: x86_64
> CPU op-mode(s): 32-bit, 64-bit
> Byte Order: Little Endian
> Vendor ID: GenuineIntel
> CPU family: 6
> Model: 26
> Model name: Intel(R) Xeon(R) CPU W3550
> @ 3.07GHz
> )
>
>
> ---
> Dr David Holec
> Computational Materials Science group
> Department of Materials Science
> Montanuniversität Leoben
>
>
>
> Franz-Josef-Strasse 18, A-8700 Leoben, Austria
> tel. +43-(0)3842-4024211
> fax. +43-(0)3842-4024202
> materials.unileoben.ac.at
> cms.unileoben.ac.at
> ________________________________
> WHERE RESEARCH MEETS FUTURE
>
>
> On Wed, 24 Nov 2021 at 08:27, Pavel Ondračka
> <pavel.ondracka at email.cz> wrote:
> > Hi David,
> >
> > as you said it works for you, so feel free to ignore, but I have
> > some
> > further tips if you are interested. Ubuntu switches between the
> > different blas and lapack using the "alternatives", so its
> > difficult
> > to
> > say if you actually link with the correct one.
> >
> > "ldd lapw1" in WIENROOT should show which one is actually linked,
> > what
> > you want to have is the openmp openblas
> > /usr/lib/x86_64-linux-gnu/openblas-openmp/libblas.so
> > /usr/lib/x86_64-linux-gnu/openblas-openmp/liblapack.so
> > or alternatively
> > /usr/lib/x86_64-linux-gnu/openblas-openmp/libopenblas.so
> > It looks like you linked with the pthread one. This is not a
> > problem
> > when running at single thread but at higher thread number this
> > might
> > lead to oversubscription and slowdowns as the pthreaded openblas
> > doesn't respect the OMP_NUM_THREADS set by Wien2k. So I would
> > recommend
> > to relink with the openmp OpenBLAS. BTW it is usually safer to link
> > with OpenBLAS explicitly using the -lopenblas instead of the -
> > llapack
> > -
> > lblas to be sure you don't accidentally link the netlib one
> > (libopenblas is just the libblas and libblapack provided by
> > OpenBLAS
> > merged together).
> >
> > In general easy way how to check performance is to run the serial
> > test_case from http://www.wien2k.at/reg_user/benchmark/ On modern
> > CPUs
> > (at least avx2) the runtime should be around 15-25 seconds at
> > single
> > thread.
> >
> > I see total runtime of ~18seconds on Fedora 35 with gfortran 11.2.1
> > OpenBLAS and AMD Ryzen 9 3900X 12-Core Processor.
> > Also look for the following line in test_case.output1, this is what
> > I
> > have:
> > TIME HAMILT (WALL) = 2.2, HNS = 1.7, HORB = 0.0, DIAG
> > =
> >
> > 14.0, SYNC = 0.0
> > The time in HAMILT mostly depends on you compiler and vectorizing
> > settings, while the DIAG is 99% lapack/blas related, so this can
> > help
> > with the diagnostics if things are slow.
> >
> > You might also get extra speedup of the HAMILT part by adding "-
> > DHAVE_LIBMVEC" to the Compiler options.
> >
> > Best regards
> > Pavel
> >
> > On Tue, 2021-11-23 at 11:07 +0100, David Holec wrote:
> > > Dear all,
> > >
> > > I have just spent some time making Wien2k run on my single
> > > machine
> > > running Ubuntu 20.04 with gfortran/gcc. Since I am not an expert,
> > it
> > > was a trial and error, but it seems that I found a working
> > combination
> > > (sadly, the default parameters didn't work for me). Maybe this
> > > will
> > > help someone. Here are the settings that did the job for me:
> > >
> > > M OpenMP switch: -fopenmp
> > > O Compiler options: -ffree-form -O2 -ftree-vectorize -
> > > march=native -ffree-line-length-none -ffpe-summary=none
> > > L Linker Flags: $(FOPT) -L/usr/lib/x86_64-linux-
> > > gnu
> > > P Preprocessor flags '-DParallel'
> > > R R_LIBS (LAPACK+BLAS): -lblas -llapack -lpthread
> > > F FFTW options: -DFFTW3 -DFFTW_OMP -I/usr/include
> > > FFTW-LIBS: -L/usr/lib/x86_64-linux-gnu -
> > > lfftw3
> > -
> > > lfftw3_omp
> > >
> > > where the FFTW options were:
> > >
> > > R FFTWROOT: /usr/
> > > V FFTW_VERSION: FFTW3
> > > L FFTW_LIB: lib/x86_64-linux-gnu
> > > N FFTW_LIBNAME: fftw3
> > >
> > > Compiler versions:
> > > $ gcc --version
> > > gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
> > > gfortran --version
> > > GNU Fortran (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
> > >
> > > And I used the generic lapack and openblas packages provides by
> > Ubuntu
> > > repos:
> > > liblapack-dev/focal,now 3.9.0-1build1 amd64 [installed]
> > > liblapack3/focal,now 3.9.0-1build1 amd64 [installed,automatic]
> > >
> > > liblapack64-3/focal,now 3.9.0-1build1 amd64 [installed,automatic]
> > > liblapack64-dev/focal,now 3.9.0-1build1 amd64 [installed]
> > >
> > > libblas-dev/focal,now 3.9.0-1build1 amd64 [installed]
> > > libblas3/focal,now 3.9.0-1build1 amd64 [installed,automatic]
> > > libblas64-3/focal,now 3.9.0-1build1 amd64 [installed,automatic]
> > > libblas64-dev/focal,now 3.9.0-1build1 amd64 [installed,automatic]
> > >
> > > libopenblas64-0/focal-updates,now 0.3.8+ds-1ubuntu0.20.04.1 amd64
> > > [installed]
> > > libopenblas64-0-openmp/focal-updates,now 0.3.8+ds-
> > > 1ubuntu0.20.04.1
> > > amd64 [installed]
> > > libopenblas64-0-pthread/focal-updates,now 0.3.8+ds-
> > > 1ubuntu0.20.04.1
> > > amd64 [installed,automatic]
> > >
> > > (I am not totally sure if I need all the libraries above, but
> > > certainly, with these, the compilation seems to work and I am
> > > able
> > to
> > > run SCF cycles & Telnes calculations without errors :-)
> > >
> > > All the best,
> > > David
> > > ---
> > > Dr David Holec
> > > Computational Materials Science group
> > > Department of Materials Science
> > > Montanuniversität Leoben
> > >
> > >
> > >
> > > Franz-Josef-Strasse 18, A-8700 Leoben, Austria
> > > tel. +43-(0)3842-4024211
> > > fax. +43-(0)3842-4024202
> > > materials.unileoben.ac.at
> > > cms.unileoben.ac.at
> > > ________________________________
> > > WHERE RESEARCH MEETS FUTURE
> > > _______________________________________________
> > > Wien mailing list
> > > Wien at zeus.theochem.tuwien.ac.at
> > > http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
> > > SEARCH the MAILING-LIST at:
> > >
> > http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html
> >
> > _______________________________________________
> > Wien mailing list
> > Wien at zeus.theochem.tuwien.ac.at
> > http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
> > SEARCH the MAILING-LIST at:
> > http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html
> _______________________________________________
> Wien mailing list
> Wien at zeus.theochem.tuwien.ac.at
> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
> SEARCH the MAILING-LIST at:
> http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html
More information about the Wien
mailing list