<div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif;color:#000000">I would not be so sure it is MKL. Did you check the version? Did you find a way to reproduce it with -g and/or -C?</div><div class="gmail_default" style="font-family:verdana,sans-serif;color:#000000"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:#000000">MKL is not perfect, but...</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Aug 19, 2021 at 1:14 PM Pavel Ondračka <<a href="mailto:pavel.ondracka@email.cz">pavel.ondracka@email.cz</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">BTW I did the Valgrind run and there is nothing there (I don't have the<br>
affected MKL, but either with OpenBLAS or with the Netlib LAPACK/BLAS<br>
there are no Valgrind defects at all in the Wien2k code, just some<br>
harmless leaked memory.) So yeah, confirming this is definitelly MKL.<br>
<br>
Pavel<br>
<br>
On Thu, 2021-08-19 at 06:56 -0500, Laurence Marks wrote:<br>
> A suggestion: check your mkl version, as there is a mkl bug that was<br>
> recently fixed, see<br>
> <a href="https://urldefense.com/v3/__https://community.intel.com/t5/Intel-oneAPI-Math-Kernel-Library/Problem-with-LAPACK-subroutine-ZHEEVR-input-array-quot-isuppz/td-p/1150816__;!!Dq0X2DkFhyF93HkjWTBQKhk!BG0vEdNiBkZCK3vqoyHfxw_N9B0iIybwSzUGRtERuXF8u0BRmgsH-bCEirX_5KLSo9nhrw$" rel="noreferrer" target="_blank">https://urldefense.com/v3/__https://community.intel.com/t5/Intel-oneAPI-Math-Kernel-Library/Problem-with-LAPACK-subroutine-ZHEEVR-input-array-quot-isuppz/td-p/1150816__;!!Dq0X2DkFhyF93HkjWTBQKhk!BG0vEdNiBkZCK3vqoyHfxw_N9B0iIybwSzUGRtERuXF8u0BRmgsH-bCEirX_5KLSo9nhrw$</a> <br>
> _____<br>
> Professor Laurence Marks<br>
> "Research is to see what everybody else has seen, and to think what<br>
> nobody else has thought", Albert Szent-Györgyi<br>
> <a href="http://www.numis.northwestern.edu" rel="noreferrer" target="_blank">http://www.numis.northwestern.edu</a><br>
> <br>
> On Thu, Aug 19, 2021, 06:45 Peter Blaha<br>
> <<a href="mailto:pblaha@theochem.tuwien.ac.at" target="_blank">pblaha@theochem.tuwien.ac.at</a>> wrote:<br>
> > I'm still on vacations, so cannot test myself.<br>
> > <br>
> > However, I experienced such problems before. It has to do with <br>
> > multithreading (1 thread works always fine) and the mkl routine<br>
> > zheevr.<br>
> > <br>
> > In my case I could fix the problem by enlarging the workspace<br>
> > beyond <br>
> > what the routine calculates itself. (see comment in hmsec on line<br>
> > 841).<br>
> > <br>
> > Right below, the workspace was enlarged by a factor 10, which fixed<br>
> > my <br>
> > problem. But I can easily envision that it might not be enough in<br>
> > some <br>
> > other cases.<br>
> > <br>
> > An alternative is to switch back to zheevx (commented in the code).<br>
> > <br>
> > Peter Blaha<br>
> > <br>
> > Am 18.08.2021 um 20:01 schrieb Pavel Ondračka:<br>
> > > Right, I think that the reason deallocate is failing because the<br>
> > memory<br>
> > > has been corrupted at some earlier point is quite clear, the only<br>
> > other<br>
> > > option why it should crash would be that it was not allocated at<br>
> > all,<br>
> > > which seem not to be the case here... The question is what<br>
> > corrupted<br>
> > > the memory and even more strange is why does it work if we<br>
> > > disable<br>
> > MKL<br>
> > > multithreading?<br>
> > > <br>
> > > It could indeed be that we are doing something wrong. I can<br>
> > > imagine<br>
> > the<br>
> > > memory could be corrupted in some BLAS call if the number of<br>
> > > columns/rows passed to the specific BLAS call is more than the<br>
> > actual<br>
> > > size of the matrix, than this could easily happen (and the<br>
> > > multithreading is somehow influencing what the final value of the<br>
> > > corrupted memory, and depending on the final value the deallocate<br>
> > could<br>
> > > fail or pass somehow). This should be possible to diagnose with<br>
> > > valgrind as suggested.<br>
> > > <br>
> > > Luis, can you upload the testcase somewhere, or recompile with<br>
> > > debuginfo as suggested by Laurence earlier, run "valgrind --<br>
> > > track-<br>
> > > origins=yes lapwso lapwso.def" and send the output? Just be<br>
> > > warned,<br>
> > > there is a massive slowdown with valgrind (up to 100x) and the<br>
> > logfile<br>
> > > can get very large.<br>
> > > <br>
> > > Best regards<br>
> > > Pavel<br>
> > > <br>
> > > <br>
> > > On Wed, 2021-08-18 at 12:10 -0500, Laurence Marks wrote:<br>
> > > > Correction, I was looking at an older modules.F. It looks like<br>
> > > > it<br>
> > > > should be<br>
> > > > <br>
> > > > DEALLOCATE(vect,stat=IV) ; if(IV .ne. 0)write(*,*)IV<br>
> > > > <br>
> > > > <br>
> > > > On Wed, Aug 18, 2021 at 11:23 AM Laurence Marks<br>
> > > > <<a href="mailto:laurence.marks@gmail.com" target="_blank">laurence.marks@gmail.com</a>> wrote:<br>
> > > > > I do wonder about this. I suggest editing module.F and<br>
> > > > > changing<br>
> > > > > lines 118 and 119 to<br>
> > > > > DEALLOCATE(en,stat=Ien) ; if(Ien .ne. 0)write(*,*)'Err<br>
> > > > > en<br>
> > > > > ',ien<br>
> > > > > DEALLOCATE(vnorm,stat=Ivn ; ) if(Ivn .ne.<br>
> > > > > 0)write(*,*)'Err<br>
> > > > > vnorm ',Ivn<br>
> > > > > <br>
> > > > > There is every chance that the bug is not in those lines, but<br>
> > > > > somewhere completely different. SIGSEV often means that the<br>
> > > > > code<br>
> > > > > has been overwritten, for instance arrays going out of<br>
> > > > > bounds.<br>
> > > > > <br>
> > > > > You can also recompile with -g (don't change other options)<br>
> > > > > added, and/or -C. Sometimes this is better. Or use other<br>
> > > > > things<br>
> > > > > like debuggers or valgrind.<br>
> > > > > <br>
> > > > > On Wed, Aug 18, 2021 at 10:47 AM Pavel Ondračka<br>
> > > > > <<a href="mailto:pavel.ondracka@email.cz" target="_blank">pavel.ondracka@email.cz</a>> wrote:<br>
> > > > > > I'm CCing the list back as the crash was now diagnosed to a<br>
> > > > > > likely<br>
> > > > > > MKL<br>
> > > > > > problem, see below for more details.<br>
> > > > > > > <br>
> > > > > > > > So just to be clear, explicitly setting<br>
> > > > > > > > OMP_STACKSIZE=1g does<br>
> > > > > > not<br>
> > > > > > > > help<br>
> > > > > > > > to solve the issue?<br>
> > > > > > > > <br>
> > > > > > > <br>
> > > > > > > Right! OMP_STACKSIZE=1g with OMP_NUM_THREADS=4 does not<br>
> > > > > > > solve<br>
> > > > > > > the<br>
> > > > > > > problem!<br>
> > > > > > > <br>
> > > > > > > > The problem is that the OpenMP code in lapwso is very<br>
> > > > > > > > simple,<br>
> > > > > > so I'm<br>
> > > > > > > > having problems seeing how it could be causing the<br>
> > > > > > > > problems.<br>
> > > > > > > > <br>
> > > > > > > > Could you also try to see what happens if run with:<br>
> > > > > > > > OMP_NUM_THREADS=1<br>
> > > > > > > > MKL_NUM_THREADS=4<br>
> > > > > > > > <br>
> > > > > > > <br>
> > > > > > > It does not work with these values, but I checked and it<br>
> > > > > > > works<br>
> > > > > > > reverting them:<br>
> > > > > > > OMP_NUM_THREADS=4<br>
> > > > > > > MKL_NUM_THREADS=1<br>
> > > > > > This was very helpfull and IMO points to a problem with MKL<br>
> > > > > > instead<br>
> > > > > > of<br>
> > > > > > Wien2k.<br>
> > > > > > <br>
> > > > > > Unfortunatelly setting MKL_NUM_THREADS=1 globally will<br>
> > > > > > reduce<br>
> > the<br>
> > > > > > OpenMP performance, mostly in lapw1 but also at other<br>
> > > > > > places. So<br>
> > > > > > if<br>
> > > > > > you<br>
> > > > > > want to keep the OpenMP BLAS/lapack level parallelism you<br>
> > > > > > have<br>
> > to<br>
> > > > > > either find some MKL version that works (if you do please<br>
> > > > > > report<br>
> > > > > > it<br>
> > > > > > here), link with OpenBLAS (using it for lapwso is enough)<br>
> > > > > > or<br>
> > > > > > create<br>
> > > > > > a<br>
> > > > > > simple wrapper that sets the MKL_NUM_THREADS=1 just for<br>
> > > > > > lapwso,<br>
> > > > > > i.e.,<br>
> > > > > > rename lapwso binary in WIENROOT to lapwso_bin and create<br>
> > > > > > new<br>
> > > > > > lapwso<br>
> > > > > > file there with:<br>
> > > > > > <br>
> > > > > > #!/bin/bash<br>
> > > > > > MKL_NUM_THREADS=1 lapwso_bin $1<br>
> > > > > > <br>
> > > > > > and set it to executable with chmod +x lapwso.<br>
> > > > > > <br>
> > > > > > Or maybe MKL has a non-OpenMP version which you could link<br>
> > > > > > with<br>
> > > > > > just<br>
> > > > > > lapwso and use standard one in other parts, but dunno, I<br>
> > > > > > mostly<br>
> > > > > > use<br>
> > > > > > OpenBLAS. If you need some further help, let me know.<br>
> > > > > > <br>
> > > > > > Reporting the issue to intel could be also nice, however I<br>
> > > > > > never<br>
> > > > > > had<br>
> > > > > > any real luck there and it is also a bit problematic as you<br>
> > can't<br>
> > > > > > provide testcase due to Wien2k being proprietary code...<br>
> > > > > > <br>
> > > > > > Best regards<br>
> > > > > > Pavel<br>
> > > > > > <br>
> > > > > > > <br>
> > > > > > > > This should disable the Wien2k-specific OpenMP<br>
> > > > > > > > parallelism<br>
> > > > > > > > but<br>
> > > > > > still<br>
> > > > > > > > keep the rest of paralellism at the BLAS/lapack level.<br>
> > > > > > > > <br>
> > > > > > > <br>
> > > > > > > So, perhaps, the problem is related to MKL!<br>
> > > > > > > <br>
> > > > > > > > Another option is that something is going wrong before<br>
> > > > > > > > lapwso<br>
> > > > > > and the<br>
> > > > > > > > lapwso crash is just the symptom. What happens if you<br>
> > > > > > > > run<br>
> > > > > > everything<br>
> > > > > > > > up<br>
> > > > > > > > to lapwso without OpenMP (OMP_NUM_THREADS=1) and than<br>
> > > > > > > > enable<br>
> > > > > > > > it<br>
> > > > > > just<br>
> > > > > > > > for lapwso?<br>
> > > > > > > > <br>
> > > > > > > <br>
> > > > > > > If I run lapw0 and lapw1 with OMP_NUM_THREADS=4 and then<br>
> > > > > > > change<br>
> > > > > > it to 1<br>
> > > > > > > just before lapwso, it works.<br>
> > > > > > > If I do the opposite, starting with OMP_NUM_THREADS=1 and<br>
> > > > > > > then<br>
> > > > > > change<br>
> > > > > > > it to 4 just before lapwso, it does not work.<br>
> > > > > > > So I believe that the problem is really at lapwso.<br>
> > > > > > > <br>
> > > > > > > If you need more information, please, let me know!<br>
> > > > > > > All the best,<br>
> > > > > > > Luis<br>
> > > > > > <br>
> > > > > > _______________________________________________<br>
> > > > > > Wien mailing list<br>
> > > > > > <a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a><br>
> > > > > > <br>
> > <a href="https://urldefense.com/v3/__http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien__;!!Dq0X2DkFhyF93HkjWTBQKhk!H_VXJmyf6v2ZSCmTICvdVDv1QuKxPqCDcjbbytr7Fh51-KF5rv8A2uvyMlW3x3YA4jSb3A$" rel="noreferrer" target="_blank">https://urldefense.com/v3/__http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien__;!!Dq0X2DkFhyF93HkjWTBQKhk!H_VXJmyf6v2ZSCmTICvdVDv1QuKxPqCDcjbbytr7Fh51-KF5rv8A2uvyMlW3x3YA4jSb3A$</a><br>
> > > > > > <br>
> > > > > > SEARCH the MAILING-LIST at:<br>
> > > > > > <br>
> > <a href="https://urldefense.com/v3/__http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html__;!!Dq0X2DkFhyF93HkjWTBQKhk!H_VXJmyf6v2ZSCmTICvdVDv1QuKxPqCDcjbbytr7Fh51-KF5rv8A2uvyMlW3x3aDFmAN4g$" rel="noreferrer" target="_blank">https://urldefense.com/v3/__http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html__;!!Dq0X2DkFhyF93HkjWTBQKhk!H_VXJmyf6v2ZSCmTICvdVDv1QuKxPqCDcjbbytr7Fh51-KF5rv8A2uvyMlW3x3aDFmAN4g$</a><br>
> > > > > > <br>
> > > > > <br>
> > > > > -- <br>
> > > > > Professor Laurence Marks<br>
> > > > > Department of Materials Science and Engineering<br>
> > > > > Northwestern University<br>
> > > > > <a href="http://www.numis.northwestern.edu" rel="noreferrer" target="_blank">http://www.numis.northwestern.edu</a><br>
> > > > > "Research is to see what everybody else has seen, and to<br>
> > > > > think<br>
> > what<br>
> > > > > nobody else has thought" Albert Szent-Györgyi<br>
> > > > <br>
> > > > _______________________________________________<br>
> > > > Wien mailing list<br>
> > > > <a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a><br>
> > > > <br>
> > <a href="https://urldefense.com/v3/__http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien__;!!Dq0X2DkFhyF93HkjWTBQKhk!HF2PQeyPOOPTweMAtKX-0JDVvq33-IxKuq0rp4xRH5r9Zgxq_eFDeApwqHjuW4E5AcHVtA$" rel="noreferrer" target="_blank">https://urldefense.com/v3/__http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien__;!!Dq0X2DkFhyF93HkjWTBQKhk!HF2PQeyPOOPTweMAtKX-0JDVvq33-IxKuq0rp4xRH5r9Zgxq_eFDeApwqHjuW4E5AcHVtA$</a><br>
> > <br>
> > > > SEARCH the MAILING-LIST at:<br>
> > > > <br>
> > <a href="https://urldefense.com/v3/__http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html__;!!Dq0X2DkFhyF93HkjWTBQKhk!HF2PQeyPOOPTweMAtKX-0JDVvq33-IxKuq0rp4xRH5r9Zgxq_eFDeApwqHjuW4GA_JKurA$" rel="noreferrer" target="_blank">https://urldefense.com/v3/__http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html__;!!Dq0X2DkFhyF93HkjWTBQKhk!HF2PQeyPOOPTweMAtKX-0JDVvq33-IxKuq0rp4xRH5r9Zgxq_eFDeApwqHjuW4GA_JKurA$</a><br>
> > <br>
> > > <br>
> > > _______________________________________________<br>
> > > Wien mailing list<br>
> > > <a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a><br>
> > > <br>
> > <a href="https://urldefense.com/v3/__http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien__;!!Dq0X2DkFhyF93HkjWTBQKhk!HF2PQeyPOOPTweMAtKX-0JDVvq33-IxKuq0rp4xRH5r9Zgxq_eFDeApwqHjuW4E5AcHVtA$" rel="noreferrer" target="_blank">https://urldefense.com/v3/__http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien__;!!Dq0X2DkFhyF93HkjWTBQKhk!HF2PQeyPOOPTweMAtKX-0JDVvq33-IxKuq0rp4xRH5r9Zgxq_eFDeApwqHjuW4E5AcHVtA$</a><br>
> > <br>
> > > SEARCH the MAILING-LIST at: <br>
> > <a href="https://urldefense.com/v3/__http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html__;!!Dq0X2DkFhyF93HkjWTBQKhk!HF2PQeyPOOPTweMAtKX-0JDVvq33-IxKuq0rp4xRH5r9Zgxq_eFDeApwqHjuW4GA_JKurA$" rel="noreferrer" target="_blank">https://urldefense.com/v3/__http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html__;!!Dq0X2DkFhyF93HkjWTBQKhk!HF2PQeyPOOPTweMAtKX-0JDVvq33-IxKuq0rp4xRH5r9Zgxq_eFDeApwqHjuW4GA_JKurA$</a><br>
> > <br>
> > <br>
> > _______________________________________________<br>
> > Wien mailing list<br>
> > <a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a><br>
> > <a href="https://urldefense.com/v3/__http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien__;!!Dq0X2DkFhyF93HkjWTBQKhk!BG0vEdNiBkZCK3vqoyHfxw_N9B0iIybwSzUGRtERuXF8u0BRmgsH-bCEirX_5KLjCONQNA$" rel="noreferrer" target="_blank">https://urldefense.com/v3/__http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien__;!!Dq0X2DkFhyF93HkjWTBQKhk!BG0vEdNiBkZCK3vqoyHfxw_N9B0iIybwSzUGRtERuXF8u0BRmgsH-bCEirX_5KLjCONQNA$</a> <br>
> > SEARCH the MAILING-LIST at: <br>
> > <a href="https://urldefense.com/v3/__http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html__;!!Dq0X2DkFhyF93HkjWTBQKhk!BG0vEdNiBkZCK3vqoyHfxw_N9B0iIybwSzUGRtERuXF8u0BRmgsH-bCEirX_5KL92k3Few$" rel="noreferrer" target="_blank">https://urldefense.com/v3/__http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html__;!!Dq0X2DkFhyF93HkjWTBQKhk!BG0vEdNiBkZCK3vqoyHfxw_N9B0iIybwSzUGRtERuXF8u0BRmgsH-bCEirX_5KL92k3Few$</a> <br>
<br>
<br>
_______________________________________________<br>
Wien mailing list<br>
<a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.at</a><br>
<a href="https://urldefense.com/v3/__http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien__;!!Dq0X2DkFhyF93HkjWTBQKhk!BG0vEdNiBkZCK3vqoyHfxw_N9B0iIybwSzUGRtERuXF8u0BRmgsH-bCEirX_5KLjCONQNA$" rel="noreferrer" target="_blank">https://urldefense.com/v3/__http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien__;!!Dq0X2DkFhyF93HkjWTBQKhk!BG0vEdNiBkZCK3vqoyHfxw_N9B0iIybwSzUGRtERuXF8u0BRmgsH-bCEirX_5KLjCONQNA$</a> <br>
SEARCH the MAILING-LIST at: <a href="https://urldefense.com/v3/__http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html__;!!Dq0X2DkFhyF93HkjWTBQKhk!BG0vEdNiBkZCK3vqoyHfxw_N9B0iIybwSzUGRtERuXF8u0BRmgsH-bCEirX_5KL92k3Few$" rel="noreferrer" target="_blank">https://urldefense.com/v3/__http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html__;!!Dq0X2DkFhyF93HkjWTBQKhk!BG0vEdNiBkZCK3vqoyHfxw_N9B0iIybwSzUGRtERuXF8u0BRmgsH-bCEirX_5KL92k3Few$</a> <br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr">Professor Laurence Marks<br>Department of Materials Science and Engineering<br>Northwestern University<br><a href="http://www.numis.northwestern.edu/" target="_blank">www.numis.northwestern.edu</a><div>"Research is to see what everybody else has seen, and to think what nobody else has thought" Albert <span style="font-family:Arial,Helvetica,sans-serif;font-size:12.8px">Szent-</span><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)">Györgyi</span></div></div></div>