<html dir="ltr"><head></head><body style="text-align:left; direction:ltr;"><div>Dear prof. Marks,</div><div>sorry for getting to this so late. Unfortunately I'm still quite unsure how to use you testcase. It sort of works, although takes a lot of iterations to converge. So I've rather tried to look at the generated code.</div><div><br></div><div>I've looked at one specific place with one randomly chosen kahan loop at the charge.f:142-148</div><div>The ifort (IFORT) 17.0.1 20161005 with the default Wien2k flags produces code like this:</div><div>I've added some comments so hopefully this is understandable</div><div>-------snip------</div><div> .loc 1 144 is_stmt 1</div><div> movsd -8(%rdx,%rcx,8), %xmm2 #copy the F(J) value to %xmm2 register</div><div>..LN397:</div><div> .loc 1 145 is_stmt 1</div><div> movaps %xmm1, %xmm3 #copy the sum value to %xmm3 register </div><div>..LN398:</div><div> .loc 1 148 is_stmt 1</div><div> incq %rcx # this is likely the loop counter scheduled here for better latency</div><div>..LN399:</div><div> .loc 1 144 is_stmt 1</div><div> subsd %xmm4, %xmm2 #%xmm2 now contains F(J) - C, i.e. Y</div><div>..LN400:</div><div> .loc 1 145 is_stmt 1</div><div> addsd %xmm2, %xmm3 #%xmm3 now contains sum + Y, i.e. T, so far so good</div><div>..LN401:</div><div> .loc 1 146 is_stmt 1</div><div> movaps %xmm3, %xmm4 #copy %xmm3 (T) to %xmm4 (the original C)</div><div>..LN402:</div><div> subsd %xmm2, %xmm4 # %xmm4 now contains T-Y!</div><div>..LN403:</div><div> subsd %xmm1, %xmm4 #only now is the sum substracted from %xmm4!</div><div>------snip---------</div><div><br></div><div><div>The last two instructions are IMO (but I'm no assembly expert) problematic. The parentheses are not honored and the line charge.f146 seems to be evaluated as C=(T-Y)-sum.</div></div><div><div><br></div></div><div><br></div><div>This is how the assembly looks with the gfortran, and default flags:</div><div><div>-----snip-----</div><div><div> .loc 1 144 0</div><div> movsd 24(%rax), %xmm2 # copy F(J) to %xmm2</div><div> .loc 1 145 0</div><div> movapd %xmm3, %xmm4 # copy sum from %xmm3 to %xmm4</div><div> addq $8, %rax #this is likely the loop counter scheduled here for better latency</div><div> .loc 1 144 0</div><div> subsd %xmm1, %xmm2 # F(J) - C, i.e. Y, store in %xmm3</div><div>.LVL8:</div><div> .loc 1 145 0</div><div> addsd %xmm2, %xmm4 # SUM + Y, i.e. T, store in %xmm4</div><div>.LVL9:</div><div> .loc 1 146 0</div><div> movapd %xmm4, %xmm1 # move T to %xmm1</div><div>.LVL10:</div><div> subsd %xmm3, %xmm1 # %xmm1 is now T - SUM</div><div> .loc 1 145 0</div><div> movapd %xmm4, %xmm3 # SUM=T</div><div> .loc 1 146 0</div><div> subsd %xmm2, %xmm1 # %xmm1 (which ATM contains T-SUM) - Y</div><div>-----snip-----</div></div></div><div></div><div><br></div><div>In other words the gfortran does it properly, and is unaffected this time. ;-) I can force similar behavior if I add the -ffast-math switch which force usafe math optimizations (but otherwise even the -O3 -march=native level is unaffected).</div><div><br></div><div>I'm unsure if this is ifort bug or if the default Wien2k flags are just too aggressive? Unfortunately I'm no fortran specification expert. On the other hand I was expecting the instructions to be optimized away. They are not, but the order is wrong, so this might be just a bug?</div><div><br></div><div>BTW I've already seen some signs that the ifort is being ifort quite aggressive before when I experimented with the libmvec to speed up the Hamilt in lapw1. I need to use the -ffast-math switch to achieve similar performance levels as ifort (and the gfortran with -ffast-math, although less precise, produces actually results which are more similar to ifort with the default wien2k flags.)</div><div><br></div><div>Hope this helps a little, if you need more help let me know.</div><div>Best regards</div><div>Pavel</div><div><br></div><div>Laurence Marks píše v Čt 09. 08. 2018 v 08:11 -0500:</div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div dir="ltr">I changed the version to include a pre-converged MSR1<div><br></div><div>A fresh pair of eyes would be useful. The minimum is somewhere in the range for atom 7 z 0.2610-0.2616, but it is very ill-conditioned and noisy. Why is unclear.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 9, 2018 at 7:11 AM, Laurence Marks <span dir="ltr"><<a href="mailto:L-marks@northwestern.edu" target="_blank">L-marks@northwestern.edu</a>></span> wrote:<br><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div dir="ltr">Test case is not a simple question, as the Kahan summations in charge.f are used in lapw0/1/2 mixer and a few other places. I have noticed minor changes in energies (1E-5 Ryd) with preventing ifort from optimizing charge.f. However, this is very tricky as ifort does 80 bit operations and only later truncates to 64, so it could be that without optimization the results are less accurate. In an ideal world the compiler will use 80 bit carries and not optimize out the summation.<div><br></div><div>If useful, a nasty ill-conditioned case is at<br><div><a href="https://drive.google.com/file/d/18xlI3-qf4RKOS8mWdR38bfo4a9Yq_t4u/view?usp=sharing" target="_blank">https://drive.google.com/file/<wbr>d/18xlI3-qf4RKOS8mWdR38bfo4a9Y<wbr>q_t4u/view?usp=sharing</a><br></div><div><br></div></div><div>You will need to do lstart/dstart then a few (15-25) MSR1 before switching to MSR1a. The convergence is somewhat random, some of this may be in the mixer and some elsewhere. By ill-conditioned I mean that if you first do 24 MSR1 versus 25, the number of iterations to convergence, or even whether it converges will be different. Sometimes two runs from the same starting points converge (or not) in a different number of iterations.</div><div><br></div><div>Where the ill-conditioning comes from is unclear to me.</div><div><br></div></div><div class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">On Thu, Aug 9, 2018 at 3:21 AM, Pavel Ondračka <span dir="ltr"><<a href="mailto:pavel.ondracka@email.cz" target="_blank">pavel.ondracka@email.cz</a>></span> wrote:<br><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex">I can look at the gfortran, what is your testcase?<br>
<br>
I tried to take a quick look with the full mixer using one random TiO2<br>
case. I put a breakpoint after some random Kahan sum (specifically this<br>
was at charge.f:150 in Wien2k 18.2) and I looked for the differences<br>
between O0 and O2. I was actually looking for small differences, but<br>
the value of sum was 0 with -O2 vs 739.29 with -O0!<br>
<br>
Hence in this case it looks like either the different optimization<br>
levels influence the program flow, or the optimizations caused the<br>
shift of the breakpoint to some other place. <br>
<br>
It might also be possible that this is a gdb problem since there is a<br>
lot of <br>
** On entry to DHSEQR parameter number 4 had an illegal value<br>
** On entry to DGEBAL parameter number 3 had an illegal value<br>
** On entry to DGEHRD parameter number 2 had an illegal value<br>
spam which I have no idea about and <error reading variable ....<br>
(access outside bounds of object referenced via synthetic pointer)><br>
<br>
BTW valgrind is also not happy with the mixer (even at -O0 there are<br>
lot of "Use of uninitialised value ... and On entry to DHSEQR parameter<br>
number 4 had an illegal value )<br>
<br>
If you can produce a simple testcase, I'd be happy to look into the<br>
Kahan sum problem, but at the moment I can't reproduce with the full<br>
mixer due to the aforementioned problems.<br>
<br>
Best regards<br>
Pavel<br>
<br>
Laurence Marks píše v St 08. 08. 2018 v 11:44 -0500:<br>
> I am testing adding the compiler directive !DEC$ NOOPTIMIZE to the<br>
> Kahan summations in charge.f in order to prevent ifort from<br>
> optimizing the summation away. It seems to help.<br>
> <br>
> Does anyone know if there are equivalents in gfortran or other<br>
> compilers? (I can't find anything for gfortran.)<br>
> <br>
> N.B., if anyone has experience with directives and wants to suggest<br>
> others that may be faster but will avoid optimizing away the<br>
> summation I am open to suggestions.<br>
> <br>
> _______________________________________________<br>
> Wien mailing list<br>
> <a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.a<wbr>t</a><br>
> <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__zeus.theochem.tuwien.ac.at_mailman_listinfo_wien&d=DwIGaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=U_T4PL6jwANfAy4rnxTj8IUxm818jnvqKFdqWLwmqg0&m=QyQYMuoW6bN9WUM8q9rgMGkZX1qo4mhebpNwy6CgZYg&s=AO9cbzTkaL7wjby4sVvXjic9gjJNqjr4Ok0j7lcCehA&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.<wbr>com/v2/url?u=http-3A__zeus.the<wbr>ochem.tuwien.ac.at_mailman_lis<wbr>tinfo_wien&d=DwIGaQ&c=yHlS04Hh<wbr>Braes5BQ9ueu5zKhE7rtNXt_<wbr>d012z2PA6ws&r=U_T4PL6jwANfAy4r<wbr>nxTj8IUxm818jnvqKFdqWLwmqg0&m=<wbr>QyQYMuoW6bN9WUM8q9rgMGkZX1qo4m<wbr>hebpNwy6CgZYg&s=AO9cbzTkaL7wjb<wbr>y4sVvXjic9gjJNqjr4Ok0j7lcCehA&<wbr>e=</a><br>
> SEARCH the MAILING-LIST at: <br>
> <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mail-2Darchive.com_wien-40zeus.theochem.tuwien.ac.at_index.html&d=DwIGaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=U_T4PL6jwANfAy4rnxTj8IUxm818jnvqKFdqWLwmqg0&m=QyQYMuoW6bN9WUM8q9rgMGkZX1qo4mhebpNwy6CgZYg&s=JEkVkLlljxw4YibMTPxypqQqbu7L_RFseDpnQ2k4LC8&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.<wbr>com/v2/url?u=http-3A__www.mail<wbr>-2Darchive.com_wien-40zeus.<wbr>theochem.tuwien.ac.at_index.<wbr>html&d=DwIGaQ&c=yHlS04HhBraes5<wbr>BQ9ueu5zKhE7rtNXt_d012z2PA6ws&<wbr>r=U_T4PL6jwANfAy4rnxTj8IUxm818<wbr>jnvqKFdqWLwmqg0&m=QyQYMuoW6bN9<wbr>WUM8q9rgMGkZX1qo4mhebpNwy6CgZY<wbr>g&s=JEkVkLlljxw4YibMTPxypqQqbu<wbr>7L_RFseDpnQ2k4LC8&e=</a><br>
<br>
_______________________________________________<br>
Wien mailing list<br>
<a href="mailto:Wien@zeus.theochem.tuwien.ac.at" target="_blank">Wien@zeus.theochem.tuwien.ac.a<wbr>t</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__zeus.theochem.tuwien.ac.at_mailman_listinfo_wien&d=DwIGaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=U_T4PL6jwANfAy4rnxTj8IUxm818jnvqKFdqWLwmqg0&m=QyQYMuoW6bN9WUM8q9rgMGkZX1qo4mhebpNwy6CgZYg&s=AO9cbzTkaL7wjby4sVvXjic9gjJNqjr4Ok0j7lcCehA&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.<wbr>com/v2/url?u=http-3A__zeus.the<wbr>ochem.tuwien.ac.at_mailman_lis<wbr>tinfo_wien&d=DwIGaQ&c=yHlS04Hh<wbr>Braes5BQ9ueu5zKhE7rtNXt_<wbr>d012z2PA6ws&r=U_T4PL6jwANfAy4r<wbr>nxTj8IUxm818jnvqKFdqWLwmqg0&m=<wbr>QyQYMuoW6bN9WUM8q9rgMGkZX1qo4m<wbr>hebpNwy6CgZYg&s=AO9cbzTkaL7wjb<wbr>y4sVvXjic9gjJNqjr4Ok0j7lcCehA&<wbr>e=</a><br>
SEARCH the MAILING-LIST at: <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mail-2Darchive.com_wien-40zeus.theochem.tuwien.ac.at_index.html&d=DwIGaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=U_T4PL6jwANfAy4rnxTj8IUxm818jnvqKFdqWLwmqg0&m=QyQYMuoW6bN9WUM8q9rgMGkZX1qo4mhebpNwy6CgZYg&s=JEkVkLlljxw4YibMTPxypqQqbu7L_RFseDpnQ2k4LC8&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.<wbr>com/v2/url?u=http-3A__www.mail<wbr>-2Darchive.com_wien-40zeus.<wbr>theochem.tuwien.ac.at_index.<wbr>html&d=DwIGaQ&c=yHlS04HhBraes5<wbr>BQ9ueu5zKhE7rtNXt_d012z2PA6ws&<wbr>r=U_T4PL6jwANfAy4rnxTj8IUxm818<wbr>jnvqKFdqWLwmqg0&m=QyQYMuoW6bN9<wbr>WUM8q9rgMGkZX1qo4mhebpNwy6CgZY<wbr>g&s=JEkVkLlljxw4YibMTPxypqQqbu<wbr>7L_RFseDpnQ2k4LC8&e=</a><br>
</blockquote></div><br><br clear="all"><div><br></div></div></div>-- <br><div class="m_6932708257531447663gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><span style="font-size:12.8px">Professor Laurence Marks</span><br></div><div dir="ltr"><span class=""><span style="font-size:12.8px">"Research is to see what everybody else has seen, and to think what nobody else has thought", </span><span style="font-size:12.8px">Albert Szent-Gyorgi</span><br></span><a href="http://www.numis.northwestern.edu" target="_blank">www.numis.northwestern.edu</a> ; <span style="font-size:12.8px">C<wbr>orrosion in 4D: </span><a href="http://MURI4D.numis.northwestern.edu" style="font-size:12.8px" target="_blank">MURI4D.numis.northwestern.edu</a><div><span style="font-size:12.8px">Partner of the CFW 100% program for gender equity, </span><a href="http://www.cfw.org/100-percent" style="font-size:12.8px" target="_blank">www.cfw.org/100-<wbr>percent</a></div><div>Co-Editor, Acta Cryst A</div></div></div></div></div></div></div></div></div></div></div>
</div>
</blockquote></div><br><br clear="all"><div><br></div><pre>_______________________________________________</pre><pre>Wien mailing list</pre><pre><a href="mailto:Wien@zeus.theochem.tuwien.ac.at">Wien@zeus.theochem.tuwien.ac.at</a></pre><pre><a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a></pre><pre>SEARCH the MAILING-LIST at: <a href="http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html">http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html</a></pre><pre><br></pre></div></blockquote></body></html>