[Wien] Compiling Wien2k 21.1 on Ubuntu 20.04 with gfortran
Pavel Ondračka
pavel.ondracka at email.cz
Wed Nov 24 14:43:53 CET 2021
Dear David,
nice, ~30 seconds instead of ~150 :-)
BTW is this already with "-DHAVE_LIBMVEC" in compiler options?
For your real workflow you might also try to experiment with the number
of threads vs number of k-points in parallel (now it seems you are
already running at 4 threads for the test_case that has only 1 k-
point), but for small cases with lots of k-points I would expect that
k-point parallelization would be the best.
Now that you link with the OpenMP-threaded OpenBLAS, it is important
that the number of k-points run in parallel times number of threads
allowed for lapw1/lapw2 does not exceed the total number of cores. It
seems your environment already has OMP_NUM_THREADS set to 4 (at least),
so you need to set the OpenMP threading explicitly for Wien2k.
Specifically try this .machines file (k-point parallel in lapw1/lapw2 +
OpenMP elsewhere, assuming Intel(R) Xeon(R) CPU W3550 should have 4
physical cores) with run_lapw -p for your standard use-case
-----------
1:localhost
1:localhost
1:localhost
1:localhost
omp_global:4
omp_lapw1:1
omp_lapw2:1
------------
or alternativelly (two k-points and two threads in lapw1+2)
-------------
1:localhost
1:localhost
omp_global:4
omp_lapw1:2
omp_lapw2:2
-------------
Best regards
Pavel
On Wed, 2021-11-24 at 13:55 +0100, David Holec wrote:
> Dear Pavel,
>
> Many thanks again for your patience and guidance. With the libopenblas-
> openmp-dev package it seems to work well!
>
> $ ldd lapw1
> linux-vdso.so.1 (0x00007ffca83d8000)
> libopenblas.so.0 => /lib/x86_64-linux-gnu/libopenblas.so.0
> (0x000014563f924000)
> libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5
> (0x000014563f65c000)
> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
> (0x000014563f50d000)
> libmvec.so.1 => /lib/x86_64-linux-gnu/libmvec.so.1
> (0x000014563f4e1000)
> libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1
> (0x000014563f49f000)
> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> (0x000014563f47c000)
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
> (0x000014563f288000)
> /lib64/ld-linux-x86-64.so.2 (0x0000145641b2d000)
> libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0
> (0x000014563f23e000)
> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
> (0x000014563f223000)
> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
> (0x000014563f21d000)
>
> and these options in siteconfig:
> L Linker Flags: $(FOPT) -L/usr/lib/x86_64-linux-
> gnu/openblas-openmp
> R R_LIBS (LAPACK+BLAS): -lopenblas
>
> I also get better timings now (though the TIME HAMILT are slightly
> longer, but overall improvement):
> $ ../x lapw1
> STOP LAPW1 END
> 119.400u 1.937s 0:32.53 372.9% 0+0k 0+37864io 0pf+0w
> $ grep HORB *output1*
> test_case.output1: TIME HAMILT (CPU) = 17.3, HNS = 18.4,
> HORB = 0.0, DIAG = 85.0, SYNC = 0.0
> test_case.output1: TIME HAMILT (WALL) = 4.6, HNS = 5.2,
> HORB = 0.0, DIAG = 22.0, SYNC = 0.0
>
> Thanks for your help,
> David
> ---
> Dr David Holec
> Computational Materials Science group
> Department of Materials Science
> Montanuniversität Leoben
>
>
>
> Franz-Josef-Strasse 18, A-8700 Leoben, Austria
> tel. +43-(0)3842-4024211
> fax. +43-(0)3842-4024202
> materials.unileoben.ac.at
> cms.unileoben.ac.at
> ________________________________
> WHERE RESEARCH MEETS FUTURE
>
>
> On Wed, 24 Nov 2021 at 12:39, Pavel Ondračka <pavel.ondracka at email.cz>
> wrote:
> > Hi David,
> >
> > well, it is hard to say without the debug info why the OpenBLAS
> > crahes.
> > My guess is that you link with the 64bit interface, try to install
> > the
> > standard one (openblas-openmp-devel) and replace openblas64-openmp
> > with
> > openblas-openmp everywhere in you config. Also remove the -lpthread
> > (just to be safe, but in theory should not matter), it is not needed
> > with OpenMP. If it still crashes, please recompile with debug info
> > enabled (add -g to compiler options) and send me the x lapw1 output
> > via
> > PM.
> >
> > BTW my response was mostly motivated by me suspecting you actually
> > link
> > against slow netlib BLAS (which turned out to be the case) and I
> > wanted
> > to warn others in case someone in the future would be using your
> > settings as a reference :-)
> >
> > Best regards
> > Pavel
> >
> > On Wed, 2021-11-24 at 11:09 +0100, David Holec wrote:
> > > Hi Pavel,
> > >
> > > Many thanks for your insights. As you know, I am not an expert on
> > how
> > > to compile codes, for me, this is sadly a trial and error
> > adventure.
> > >
> > > I tried to compile it against the openblas library, but although
> > the
> > > compilation ends without any errors, I get a segmentation fault
> > when
> > > running lapw1 (on the test case
> > > from http://www.wien2k.at/reg_user/benchmark/). The current setting
> > > are:
> > >
> > > L Linker Flags: $(FOPT) -L/usr/lib/x86_64-linux-
> > > gnu/openblas64-openmp
> > > R R_LIBS (LAPACK+BLAS): /usr/lib/x86_64-linux-gnu/openblas64-
> > > openmp/libopenblas64.so.0 -lpthread
> > >
> > >
> > > (The rest is as I wrote in my first email.) Here is the list of
> > > linked libraries:
> > > $ ldd lapw1
> > > linux-vdso.so.1 (0x00007ffea57d6000)
> > > libopenblas64.so.0 => /lib/x86_64-linux-
> > > gnu/libopenblas64.so.0 (0x000014fe2b2e5000)
> > > libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> > > (0x000014fe2b2c2000)
> > > libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5
> > > (0x000014fe2affa000)
> > > libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
> > > (0x000014fe2aeab000)
> > > libmvec.so.1 => /lib/x86_64-linux-gnu/libmvec.so.1
> > > (0x000014fe2ae7f000)
> > > libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1
> > > (0x000014fe2ae3d000)
> > > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
> > > (0x000014fe2ac49000)
> > > /lib64/ld-linux-x86-64.so.2 (0x000014fe2d4d3000)
> > > libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0
> > > (0x000014fe2abff000)
> > > libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
> > > (0x000014fe2abe4000)
> > > libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
> > > (0x000014fe2abde000)
> > >
> > > And here is the stacking fault (it doesn't tell me anything):
> > > $x lapw1
> > >
> > > Program received signal SIGSEGV: Segmentation fault - invalid
> > memory
> > > reference.
> > >
> > > Backtrace for this error:
> > > #0 0x148fb48b8d21 in ???
> > > #1 0x148fb48b7ef5 in ???
> > > #2 0x148fb452d20f in ???
> > > #3 0x148fb54495fb in ???
> > > Segmentation fault
> > > 0.949u 0.472s 0:00.84 167.8% 0+0k 14400+10992io 31pf+0w
> > >
> > > Any idea?
> > >
> > > With the settings I reported yesterday, everything works just fine
> > > (though probably not very efficiently - but this is not a problem
> > for
> > > me as this binary is not a "production" binary on any HPC):
> > >
> > > $ ldd lapw1
> > > linux-vdso.so.1 (0x00007ffc60765000)
> > > libblas.so.3 => /lib/x86_64-linux-gnu/libblas.so.3
> > > (0x0000150cec90f000)
> > > liblapack.so.3 => /lib/x86_64-linux-gnu/liblapack.so.3
> > > (0x0000150cec26b000)
> > > libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> > > (0x0000150cec248000)
> > > libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5
> > > (0x0000150cebf80000)
> > > libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
> > > (0x0000150cebe31000)
> > > libmvec.so.1 => /lib/x86_64-linux-gnu/libmvec.so.1
> > > (0x0000150cebe05000)
> > > libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1
> > > (0x0000150cebdc1000)
> > > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
> > > (0x0000150cebbcf000)
> > > libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
> > > (0x0000150cebbb4000)
> > > /lib64/ld-linux-x86-64.so.2 (0x0000150cec9fc000)
> > > libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0
> > > (0x0000150cebb6a000)
> > > libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
> > > (0x0000150cebb64000)
> > >
> > > and:
> > >
> > > $ x lapw1
> > > STOP LAPW1 END
> > > 162.045u 0.918s 2:31.25 107.7% 0+0k 8976+37856io 35pf+0w
> > > $ grep HORB *output1*
> > > test_case.output1: TIME HAMILT (CPU) = 16.3, HNS =
> > 20.0,
> > > HORB = 0.0, DIAG = 125.9, SYNC = 0.0
> > > test_case.output1: TIME HAMILT (WALL) = 4.4, HNS =
> > 20.0,
> > > HORB = 0.0, DIAG = 126.0, SYNC = 0.0
> > >
> > > (I am using it on my 10-year old "type writer":
> > > $lscpu
> > > Architecture: x86_64
> > > CPU op-mode(s): 32-bit, 64-bit
> > > Byte Order: Little Endian
> > > Vendor ID: GenuineIntel
> > > CPU family: 6
> > > Model: 26
> > > Model name: Intel(R) Xeon(R) CPU
> > W3550
> > > @ 3.07GHz
> > > )
> > >
> > >
> > > ---
> > > Dr David Holec
> > > Computational Materials Science group
> > > Department of Materials Science
> > > Montanuniversität Leoben
> > >
> > >
> > >
> > > Franz-Josef-Strasse 18, A-8700 Leoben, Austria
> > > tel. +43-(0)3842-4024211
> > > fax. +43-(0)3842-4024202
> > > materials.unileoben.ac.at
> > > cms.unileoben.ac.at
> > > ________________________________
> > > WHERE RESEARCH MEETS FUTURE
> > >
> > >
> > > On Wed, 24 Nov 2021 at 08:27, Pavel Ondračka
> > > <pavel.ondracka at email.cz> wrote:
> > > > Hi David,
> > > >
> > > > as you said it works for you, so feel free to ignore, but I have
> > > > some
> > > > further tips if you are interested. Ubuntu switches between the
> > > > different blas and lapack using the "alternatives", so its
> > > > difficult
> > > > to
> > > > say if you actually link with the correct one.
> > > >
> > > > "ldd lapw1" in WIENROOT should show which one is actually linked,
> > > > what
> > > > you want to have is the openmp openblas
> > > > /usr/lib/x86_64-linux-gnu/openblas-openmp/libblas.so
> > > > /usr/lib/x86_64-linux-gnu/openblas-openmp/liblapack.so
> > > > or alternatively
> > > > /usr/lib/x86_64-linux-gnu/openblas-openmp/libopenblas.so
> > > > It looks like you linked with the pthread one. This is not a
> > > > problem
> > > > when running at single thread but at higher thread number this
> > > > might
> > > > lead to oversubscription and slowdowns as the pthreaded openblas
> > > > doesn't respect the OMP_NUM_THREADS set by Wien2k. So I would
> > > > recommend
> > > > to relink with the openmp OpenBLAS. BTW it is usually safer to
> > link
> > > > with OpenBLAS explicitly using the -lopenblas instead of the -
> > > > llapack
> > > > -
> > > > lblas to be sure you don't accidentally link the netlib one
> > > > (libopenblas is just the libblas and libblapack provided by
> > > > OpenBLAS
> > > > merged together).
> > > >
> > > > In general easy way how to check performance is to run the serial
> > > > test_case from http://www.wien2k.at/reg_user/benchmark/ On modern
> > > > CPUs
> > > > (at least avx2) the runtime should be around 15-25 seconds at
> > > > single
> > > > thread.
> > > >
> > > > I see total runtime of ~18seconds on Fedora 35 with gfortran
> > 11.2.1
> > > > OpenBLAS and AMD Ryzen 9 3900X 12-Core Processor.
> > > > Also look for the following line in test_case.output1, this is
> > what
> > > > I
> > > > have:
> > > > TIME HAMILT (WALL) = 2.2, HNS = 1.7, HORB = 0.0, DIAG
> > > > =
> > > >
> > > > 14.0, SYNC = 0.0
> > > > The time in HAMILT mostly depends on you compiler and vectorizing
> > > > settings, while the DIAG is 99% lapack/blas related, so this can
> > > > help
> > > > with the diagnostics if things are slow.
> > > >
> > > > You might also get extra speedup of the HAMILT part by adding "-
> > > > DHAVE_LIBMVEC" to the Compiler options.
> > > >
> > > > Best regards
> > > > Pavel
> > > >
> > > > On Tue, 2021-11-23 at 11:07 +0100, David Holec wrote:
> > > > > Dear all,
> > > > >
> > > > > I have just spent some time making Wien2k run on my single
> > > > > machine
> > > > > running Ubuntu 20.04 with gfortran/gcc. Since I am not an
> > expert,
> > > > it
> > > > > was a trial and error, but it seems that I found a working
> > > > combination
> > > > > (sadly, the default parameters didn't work for me). Maybe this
> > > > > will
> > > > > help someone. Here are the settings that did the job for me:
> > > > >
> > > > > M OpenMP switch: -fopenmp
> > > > > O Compiler options: -ffree-form -O2 -ftree-vectorize
> > -
> > > > > march=native -ffree-line-length-none -ffpe-summary=none
> > > > > L Linker Flags: $(FOPT) -L/usr/lib/x86_64-linux-
> > > > > gnu
> > > > > P Preprocessor flags '-DParallel'
> > > > > R R_LIBS (LAPACK+BLAS): -lblas -llapack -lpthread
> > > > > F FFTW options: -DFFTW3 -DFFTW_OMP -
> > I/usr/include
> > > > > FFTW-LIBS: -L/usr/lib/x86_64-linux-gnu -
> > > > > lfftw3
> > > > -
> > > > > lfftw3_omp
> > > > >
> > > > > where the FFTW options were:
> > > > >
> > > > > R FFTWROOT: /usr/
> > > > > V FFTW_VERSION: FFTW3
> > > > > L FFTW_LIB: lib/x86_64-linux-gnu
> > > > > N FFTW_LIBNAME: fftw3
> > > > >
> > > > > Compiler versions:
> > > > > $ gcc --version
> > > > > gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
> > > > > gfortran --version
> > > > > GNU Fortran (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
> > > > >
> > > > > And I used the generic lapack and openblas packages provides by
> > > > Ubuntu
> > > > > repos:
> > > > > liblapack-dev/focal,now 3.9.0-1build1 amd64 [installed]
> > > > > liblapack3/focal,now 3.9.0-1build1 amd64 [installed,automatic]
> > > > >
> > > > > liblapack64-3/focal,now 3.9.0-1build1 amd64
> > [installed,automatic]
> > > > > liblapack64-dev/focal,now 3.9.0-1build1 amd64 [installed]
> > > > >
> > > > > libblas-dev/focal,now 3.9.0-1build1 amd64 [installed]
> > > > > libblas3/focal,now 3.9.0-1build1 amd64 [installed,automatic]
> > > > > libblas64-3/focal,now 3.9.0-1build1 amd64 [installed,automatic]
> > > > > libblas64-dev/focal,now 3.9.0-1build1 amd64
> > [installed,automatic]
> > > > >
> > > > > libopenblas64-0/focal-updates,now 0.3.8+ds-1ubuntu0.20.04.1
> > amd64
> > > > > [installed]
> > > > > libopenblas64-0-openmp/focal-updates,now 0.3.8+ds-
> > > > > 1ubuntu0.20.04.1
> > > > > amd64 [installed]
> > > > > libopenblas64-0-pthread/focal-updates,now 0.3.8+ds-
> > > > > 1ubuntu0.20.04.1
> > > > > amd64 [installed,automatic]
> > > > >
> > > > > (I am not totally sure if I need all the libraries above, but
> > > > > certainly, with these, the compilation seems to work and I am
> > > > > able
> > > > to
> > > > > run SCF cycles & Telnes calculations without errors :-)
> > > > >
> > > > > All the best,
> > > > > David
> > > > > ---
> > > > > Dr David Holec
> > > > > Computational Materials Science group
> > > > > Department of Materials Science
> > > > > Montanuniversität Leoben
> > > > >
> > > > >
> > > > >
> > > > > Franz-Josef-Strasse 18, A-8700 Leoben, Austria
> > > > > tel. +43-(0)3842-4024211
> > > > > fax. +43-(0)3842-4024202
> > > > > materials.unileoben.ac.at
> > > > > cms.unileoben.ac.at
> > > > > ________________________________
> > > > > WHERE RESEARCH MEETS FUTURE
> > > > > _______________________________________________
> > > > > Wien mailing list
> > > > > Wien at zeus.theochem.tuwien.ac.at
> > > > > http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
> > > > > SEARCH the MAILING-LIST at:
> > > > >
> > > >
> > http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html
> > > >
> > > > _______________________________________________
> > > > Wien mailing list
> > > > Wien at zeus.theochem.tuwien.ac.at
> > > > http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
> > > > SEARCH the MAILING-LIST at:
> > > >
> > http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html
> > > _______________________________________________
> > > Wien mailing list
> > > Wien at zeus.theochem.tuwien.ac.at
> > > http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
> > > SEARCH the MAILING-LIST at:
> > >
> > http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html
> >
> > _______________________________________________
> > Wien mailing list
> > Wien at zeus.theochem.tuwien.ac.at
> > http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
> > SEARCH the MAILING-LIST at:
> > http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html
> _______________________________________________
> Wien mailing list
> Wien at zeus.theochem.tuwien.ac.at
> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
> SEARCH the MAILING-LIST at:
> http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html
More information about the Wien
mailing list