<div dir="auto">I think Peter may have mispoke about the latest elpa. I believe it will run OK if you compile it (--enable-AVX512 etc) so the highest kernel is equal to the lowest instruction set you use. You may also get it to work by using their environmental variables. With the current Wien2k you cannot exploit elpa optimally if you have a heterogeneous set of nodes.<div dir="auto"><br></div><div dir="auto">I would say 30% faster comparing a 6130 to a E5-2650. However, ifort compiler switches can make a big difference, as can the mpi version.</div><div dir="auto"><br></div><div dir="auto">N.B., I can dig up my elpa compiler options later if needed. I use ifort/icc/mpiifort/mpiicc. <br><br><div data-smartmail="gmail_signature" dir="auto">_____<br>Professor Laurence Marks<br>"Research is to see what everybody else has seen, and to think what nobody else has thought", Albert Szent-Gyorgi<br><a href="http://www.numis.northwestern.edu" rel="noreferrer noreferrer" target="_blank">www.numis.northwestern.edu</a></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Wed, Feb 27, 2019, 02:50 Peter Blaha <<a href="mailto:pblaha@theochem.tuwien.ac.at" rel="noreferrer noreferrer" target="_blank">pblaha@theochem.tuwien.ac.at</a> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">We have an Intel I7-7820X CPU @ 3.60GHz with 8 cores and avx512.<br>
<br>
The testcase with OMP_NUM_THREADS=1 runs a bit faster with avx512 than <br>
with avx2, but it is a rather small effect (at least when working with <br>
this MKL_ENABLE_INSTRUCTIONS variable:<br>
----------------------avx512<br>
TIME HAMILT (CPU) = 5.1, HNS = 2.1, HORB = 0.0, <br>
DIAG = 15.3<br>
TIME HAMILT (WALL) = 5.4, HNS = 2.1, HORB = 0.0, <br>
DIAG = 15.3<br>
----------------------avx2<br>
TIME HAMILT (CPU) = 5.8, HNS = 2.5, HORB = 0.0, <br>
DIAG = 16.3<br>
TIME HAMILT (WALL) = 6.1, HNS = 2.5, HORB = 0.0, <br>
DIAG = 16.3<br>
<br>
However, when using OMP_NUM_THREADS=8, this difference is further <br>
reduced (probably due to memory bounds ?)<br>
-----------------------avx512<br>
TIME HAMILT (CPU) = 19.9, HNS = 7.7, HORB = 0.0, <br>
DIAG = 24.2<br>
TIME HAMILT (WALL) = 2.6, HNS = 1.0, HORB = 0.0, <br>
DIAG = 3.2<br>
------------------------avx2<br>
TIME HAMILT (CPU) = 20.0, HNS = 7.4, HORB = 0.0, <br>
DIAG = 27.0<br>
TIME HAMILT (WALL) = 2.6, HNS = 1.0, HORB = 0.0, <br>
DIAG = 3.5<br>
-------------------------------------------------------------------------<br>
<br>
Yes, we have the latest ELPA elpa-2018.11.001 installed. Seems to run <br>
without problems and is overall significantly better than the old ELPA), <br>
but it requires a change in the user interface. The next release of <br>
WIEN2k will have two elpa versions supported, a ELPA15 (which is in <br>
WIEN2k_18), and a new ELPA interface for elpa versions later than 2017 <br>
(this is somehow like FFTW2 and FFTW3 versions).<br>
<br>
So in essence: with the present code one cannot use ELPA-versions from <br>
2017 or later.<br>
<br>
On 2/27/19 7:34 AM, Pavel Ondračka wrote:<br>
> Dear mailing list,<br>
> <br>
> just out of curiosity has anyone any experience running Wien2k on a<br>
> AVX512 capable machine (eg. the KNL accelerators or recent Intel<br>
> skylake-avx512 CPUs)?<br>
> <br>
> Recently my cluster updated to this skylake-avx512 machines however I'm<br>
> unable to get any better performance for Wien2k. In particular MKL seem<br>
> to suck, for example in single core performance (with the serial<br>
> test_case) the eigenvalue problem is actually faster when I forbid the<br>
> usage of AVX512 instructions:<br>
> <br>
> running with MKL_VERBOSE=1 MKL_ENABLE_INSTRUCTIONS=AVX2<br>
> MKL_VERBOSE<br>
> ZHETRD(L,3481,0x2b74d8567cc0,3481,0x2b74d82121c0,0x2b74d8218e88,0x2b74e<br>
> f769b00,0x2b74ef777490,452530,0) 10.21s CNR:OFF Dyn:1 FastMM:1<br>
> TID:0 NThr:1<br>
> <br>
> with MKL_ENABLE_INSTRUCTIONS=AVX512<br>
> MKL_VERBOSE<br>
> ZHETRD(L,3481,0x2b5397c96cc0,3481,0x2b53979411c0,0x2b5397947e88,0x2b53a<br>
> ee98b00,0x2b53aeea6490,452530,0) 12.31s CNR:OFF Dyn:1 FastMM:1<br>
> TID:0 NThr:1<br>
> <br>
> This is somewhat compensated by speedups in the hamilt part (the VML<br>
> stuff and various ?GEMMs seem to be actually slightly faster), but<br>
> overall the performance is mostly the same with and without the AVX512<br>
> stuff. OpenBLAS is maybe 15% slower so not an option as well...<br>
> <br>
> Moreover for MPI version I'm not able to get a correctly working ELPA<br>
> compiled with the AVX512 support (I went for the latest elpa-<br>
> 2018.11.001 version), it just returns bogus results and diverges after<br>
> few iterations. If someone has this working I'd be really grateful for<br>
> a working configure line, and advice with which elpa and which compiler<br>
> version this was.<br>
> <br>
> Unfortunately I was not able to get any support from the cluster admins<br>
> beyond "We see a 30% per-core performance increase in average"<br>
> therefore asking here if anyone has experience with such machines.<br>
> <br>
> Any advice would be appreciated.<br>
> Best regards<br>
> Pavel<br>
> <br>
> _______________________________________________<br>
> Wien mailing list<br>
> <a href="mailto:Wien@zeus.theochem.tuwien.ac.at" rel="noreferrer noreferrer noreferrer" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a><br>
> <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__zeus.theochem.tuwien.ac.at_mailman_listinfo_wien&d=DwIGaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=U_T4PL6jwANfAy4rnxTj8IUxm818jnvqKFdqWLwmqg0&m=0vwn_c2KmvYL2EmszqmMAxn22_AHFhqVwSIMrLn_c_8&s=9rbXdyGFAJctXB2SLaOcC0V-kJ5Pi8IEjT4Rh-WXr7E&e=" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=http-3A__zeus.theochem.tuwien.ac.at_mailman_listinfo_wien&d=DwIGaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=U_T4PL6jwANfAy4rnxTj8IUxm818jnvqKFdqWLwmqg0&m=0vwn_c2KmvYL2EmszqmMAxn22_AHFhqVwSIMrLn_c_8&s=9rbXdyGFAJctXB2SLaOcC0V-kJ5Pi8IEjT4Rh-WXr7E&e=</a><br>
> SEARCH the MAILING-LIST at: <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mail-2Darchive.com_wien-40zeus.theochem.tuwien.ac.at_index.html&d=DwIGaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=U_T4PL6jwANfAy4rnxTj8IUxm818jnvqKFdqWLwmqg0&m=0vwn_c2KmvYL2EmszqmMAxn22_AHFhqVwSIMrLn_c_8&s=qjTxSMAPwx29qPYmofuPDU3WxGJX4Yw4QkCHJKo7T8g&e=" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mail-2Darchive.com_wien-40zeus.theochem.tuwien.ac.at_index.html&d=DwIGaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=U_T4PL6jwANfAy4rnxTj8IUxm818jnvqKFdqWLwmqg0&m=0vwn_c2KmvYL2EmszqmMAxn22_AHFhqVwSIMrLn_c_8&s=qjTxSMAPwx29qPYmofuPDU3WxGJX4Yw4QkCHJKo7T8g&e=</a><br>
> <br>
<br>
-- <br>
<br>
P.Blaha<br>
--------------------------------------------------------------------------<br>
Peter BLAHA, Inst.f. Materials Chemistry, TU Vienna, A-1060 Vienna<br>
Phone: +43-1-58801-165300 FAX: +43-1-58801-165982<br>
Email: <a href="mailto:blaha@theochem.tuwien.ac.at" rel="noreferrer noreferrer noreferrer" target="_blank">blaha@theochem.tuwien.ac.at</a> WIEN2k: <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.wien2k.at&d=DwIGaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=U_T4PL6jwANfAy4rnxTj8IUxm818jnvqKFdqWLwmqg0&m=0vwn_c2KmvYL2EmszqmMAxn22_AHFhqVwSIMrLn_c_8&s=TFV0KhtG7EcQlTVqkdKqOmMJVdxRAy3ZuDrld-uWvIM&e=" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=http-3A__www.wien2k.at&d=DwIGaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=U_T4PL6jwANfAy4rnxTj8IUxm818jnvqKFdqWLwmqg0&m=0vwn_c2KmvYL2EmszqmMAxn22_AHFhqVwSIMrLn_c_8&s=TFV0KhtG7EcQlTVqkdKqOmMJVdxRAy3ZuDrld-uWvIM&e=</a><br>
WWW: <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.imc.tuwien.ac.at_TC-5FBlaha&d=DwIGaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=U_T4PL6jwANfAy4rnxTj8IUxm818jnvqKFdqWLwmqg0&m=0vwn_c2KmvYL2EmszqmMAxn22_AHFhqVwSIMrLn_c_8&s=YmE7c8gn2QT2WRBkXhUey5BerwAAUH0MfBj8RNBoNNQ&e=" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=http-3A__www.imc.tuwien.ac.at_TC-5FBlaha&d=DwIGaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=U_T4PL6jwANfAy4rnxTj8IUxm818jnvqKFdqWLwmqg0&m=0vwn_c2KmvYL2EmszqmMAxn22_AHFhqVwSIMrLn_c_8&s=YmE7c8gn2QT2WRBkXhUey5BerwAAUH0MfBj8RNBoNNQ&e=</a><br>
--------------------------------------------------------------------------<br>
_______________________________________________<br>
Wien mailing list<br>
<a href="mailto:Wien@zeus.theochem.tuwien.ac.at" rel="noreferrer noreferrer noreferrer" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__zeus.theochem.tuwien.ac.at_mailman_listinfo_wien&d=DwIGaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=U_T4PL6jwANfAy4rnxTj8IUxm818jnvqKFdqWLwmqg0&m=0vwn_c2KmvYL2EmszqmMAxn22_AHFhqVwSIMrLn_c_8&s=9rbXdyGFAJctXB2SLaOcC0V-kJ5Pi8IEjT4Rh-WXr7E&e=" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=http-3A__zeus.theochem.tuwien.ac.at_mailman_listinfo_wien&d=DwIGaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=U_T4PL6jwANfAy4rnxTj8IUxm818jnvqKFdqWLwmqg0&m=0vwn_c2KmvYL2EmszqmMAxn22_AHFhqVwSIMrLn_c_8&s=9rbXdyGFAJctXB2SLaOcC0V-kJ5Pi8IEjT4Rh-WXr7E&e=</a><br>
SEARCH the MAILING-LIST at: <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mail-2Darchive.com_wien-40zeus.theochem.tuwien.ac.at_index.html&d=DwIGaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=U_T4PL6jwANfAy4rnxTj8IUxm818jnvqKFdqWLwmqg0&m=0vwn_c2KmvYL2EmszqmMAxn22_AHFhqVwSIMrLn_c_8&s=qjTxSMAPwx29qPYmofuPDU3WxGJX4Yw4QkCHJKo7T8g&e=" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mail-2Darchive.com_wien-40zeus.theochem.tuwien.ac.at_index.html&d=DwIGaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=U_T4PL6jwANfAy4rnxTj8IUxm818jnvqKFdqWLwmqg0&m=0vwn_c2KmvYL2EmszqmMAxn22_AHFhqVwSIMrLn_c_8&s=qjTxSMAPwx29qPYmofuPDU3WxGJX4Yw4QkCHJKo7T8g&e=</a><br>
</blockquote></div>