[Wien] OS bug/inconveniece with mkl thread/mpi conflicts

[Laurence Marks]

Agreed. Or use something like (in parallel_options, 1 k-point only,
could be generalized)

set a=`grep -e "1:" .machines | grep -v lapw0 | cut -f 3 -d: | cut -c 1-2`
setenv MKL_NUM_THREADS $a
setenv OMP_NUM_THREADS $a

or
set a=`grep -e $HOSTNAME $PBS_NODEFILE | wc -l`

My opinion is that this should be set at the OS level with
MKL_NUM_THREADS etc set by msub/qsub or similar. I've seen problems
due to other processes (not mine) using more threads than they should.

On Tue, Mar 15, 2011 at 10:35 AM, Peter Blaha
<pblaha at theochem.tuwien.ac.at> wrote:
> Setting OMP_NUM_THREAD lets you select to use multithreading or not.
>
> A slight modification of our example script on the WIEN2k faq page
> can even accounts for this and automatically span on eg. 8 core only
> 4 mpi processes (when OMP_NUM_THREAD=2).
>
>
> Am 15.03.2011 15:31, schrieb Laurence Marks:
>>
>> This is an informational report of an issue that can arise with Wien2k
>> leading to poor performance. I came across it when experimenting with
>> the "procs=XX" option in msub/qsub with XX=64 as an alternative to the
>> more standard "nodes=X:ppn=Y".
>>
>> In an scf cycle lapwdm and (with the next release) mixer always use
>> lapack routines (and maybe some others) which by default with mkl are
>> multithreaded and use multiple cores. For a mpi run with lapw0:
>> specified lapw0, lapw1 and lapw2 all use mpi. A issue can arise with
>> msub/qsub (and perhaps other) if the number of cores mkl uses is
>> larger than what has been allocated and there are other mpi tasks
>> running on the same node. Then the performance of lapwdm can be bad;
>> in one case rather than taking 30 seconds it took 14 minutes.
>>
>> I am not sure if this is something "fixable" in WIen2k, and there are
>> ways in qsub/msub to avoid it using nodes=X:ppn=Y. However, procs=XX
>> is "nice" as Wien2k in mpi does not really need factors of 2
>> cores/node (I've tested this) and can fit in the gaps.
>>
>> N.B., running without mpi and using nodes=X:ppn=1 could also run into
>> the same problem.
>>
>
> --
>
>                                      P.Blaha
> --------------------------------------------------------------------------
> Peter BLAHA, Inst.f. Materials Chemistry, TU Vienna, A-1060 Vienna
> Phone: +43-1-58801-15671             FAX: +43-1-58801-15698
> Email: blaha at theochem.tuwien.ac.at    WWW:
> http://info.tuwien.ac.at/theochem/
> --------------------------------------------------------------------------
> _______________________________________________
> Wien mailing list
> Wien at zeus.theochem.tuwien.ac.at
> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
>



-- 
Laurence Marks
Department of Materials Science and Engineering
MSE Rm 2036 Cook Hall
2220 N Campus Drive
Northwestern University
Evanston, IL 60208, USA
Tel: (847) 491-3996 Fax: (847) 491-7820
email: L-marks at northwestern dot edu
Web: www.numis.northwestern.edu
Chair, Commission on Electron Crystallography of IUCR
www.numis.northwestern.edu/
Electron crystallography is the branch of science that uses electron
scattering and imaging to study the structure of matter.