[Wien] Wien2k on AVX512 CPUs

[Peter Blaha]

We have an Intel I7-7820X CPU @ 3.60GHz with 8 cores and avx512.

The testcase with OMP_NUM_THREADS=1 runs a bit faster with avx512 than 
with avx2, but it is a rather small effect (at least when working with 
this MKL_ENABLE_INSTRUCTIONS variable:
----------------------avx512
        TIME HAMILT (CPU)  =     5.1, HNS =     2.1, HORB =     0.0, 
DIAG =    15.3
        TIME HAMILT (WALL) =     5.4, HNS =     2.1, HORB =     0.0, 
DIAG =    15.3
----------------------avx2
        TIME HAMILT (CPU)  =     5.8, HNS =     2.5, HORB =     0.0, 
DIAG =    16.3
        TIME HAMILT (WALL) =     6.1, HNS =     2.5, HORB =     0.0, 
DIAG =    16.3

However, when using OMP_NUM_THREADS=8, this difference is further 
reduced (probably due to memory bounds ?)
-----------------------avx512
        TIME HAMILT (CPU)  =    19.9, HNS =     7.7, HORB =     0.0, 
DIAG =    24.2
        TIME HAMILT (WALL) =     2.6, HNS =     1.0, HORB =     0.0, 
DIAG =     3.2
------------------------avx2
        TIME HAMILT (CPU)  =    20.0, HNS =     7.4, HORB =     0.0, 
DIAG =    27.0
        TIME HAMILT (WALL) =     2.6, HNS =     1.0, HORB =     0.0, 
DIAG =     3.5
-------------------------------------------------------------------------

Yes, we have the latest ELPA elpa-2018.11.001 installed. Seems to run 
without problems and is overall significantly better than the old ELPA), 
but it requires a change in the user interface. The next release of 
WIEN2k will have two elpa versions supported, a ELPA15 (which is in 
WIEN2k_18), and a new ELPA interface for elpa versions later than 2017 
(this is somehow like FFTW2 and FFTW3 versions).

So in essence: with the present code one cannot use ELPA-versions from 
2017 or later.

On 2/27/19 7:34 AM, Pavel Ondračka wrote:
> Dear mailing list,
> 
> just out of curiosity has anyone any experience running Wien2k on a
> AVX512 capable machine (eg. the KNL accelerators or recent Intel
> skylake-avx512 CPUs)?
> 
> Recently my cluster updated to this skylake-avx512 machines however I'm
> unable to get any better performance for Wien2k. In particular MKL seem
> to suck, for example in single core performance (with the serial
> test_case) the eigenvalue problem is actually faster when I forbid the
> usage of AVX512 instructions:
> 
> running with MKL_VERBOSE=1 MKL_ENABLE_INSTRUCTIONS=AVX2
> MKL_VERBOSE
> ZHETRD(L,3481,0x2b74d8567cc0,3481,0x2b74d82121c0,0x2b74d8218e88,0x2b74e
> f769b00,0x2b74ef777490,452530,0) 10.21s CNR:OFF Dyn:1 FastMM:1
> TID:0  NThr:1
> 
> with MKL_ENABLE_INSTRUCTIONS=AVX512
> MKL_VERBOSE
> ZHETRD(L,3481,0x2b5397c96cc0,3481,0x2b53979411c0,0x2b5397947e88,0x2b53a
> ee98b00,0x2b53aeea6490,452530,0) 12.31s CNR:OFF Dyn:1 FastMM:1
> TID:0  NThr:1
> 
> This is somewhat compensated by speedups in the hamilt part (the VML
> stuff and various ?GEMMs seem to be actually slightly faster), but
> overall the performance is mostly the same with and without the AVX512
> stuff. OpenBLAS is maybe 15% slower so not an option as well...
> 
> Moreover for MPI version I'm not able to get a correctly working ELPA
> compiled with the AVX512 support (I went for the latest elpa-
> 2018.11.001 version), it just returns bogus results and diverges after
> few iterations. If someone has this working I'd be really grateful for
> a working configure line, and advice with which elpa and which compiler
> version this was.
> 
> Unfortunately I was not able to get any support from the cluster admins
> beyond "We see a 30% per-core performance increase in average"
> therefore asking here if anyone has experience with such machines.
> 
> Any advice would be appreciated.
> Best regards
> Pavel
> 
> _______________________________________________
> Wien mailing list
> Wien at zeus.theochem.tuwien.ac.at
> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
> SEARCH the MAILING-LIST at:  http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html
> 

-- 

                                       P.Blaha
--------------------------------------------------------------------------
Peter BLAHA, Inst.f. Materials Chemistry, TU Vienna, A-1060 Vienna
Phone: +43-1-58801-165300             FAX: +43-1-58801-165982
Email: blaha at theochem.tuwien.ac.at    WIEN2k: http://www.wien2k.at
WWW:   http://www.imc.tuwien.ac.at/TC_Blaha
--------------------------------------------------------------------------