[Wien] OS bug/inconveniece with mkl thread/mpi conflicts

[Laurence Marks]

Even better (in my opinion this should be in /etc/bashrc)

export MKL_NUM_THREADS=1 OMP_NUM_THREADS=1
export MKL_DOMAIN_NUM_THREADS="MKL_VML=1, MKL_ALL=1"
if [ "$PBS_NODEFILE" != "" ] ; then
mkln=`grep -e $HOSTNAME $PBS_NODEFILE | wc -l`
export MKL_NUM_THREADS=$mkln OMP_NUM_THREADS=$mkln
export MKL_DYNAMIC=FALSE
export MKL_DOMAIN_NUM_THREADS="MKL_VML=$mkln, MKL_ALL=$mkln"
fi

Edit as appropriate for other shells.

On Tue, Mar 15, 2011 at 10:59 AM, Laurence Marks
<L-marks at northwestern.edu> wrote:
> Agreed. Or use something like (in parallel_options, 1 k-point only,
> could be generalized)
>
> set a=`grep -e "1:" .machines | grep -v lapw0 | cut -f 3 -d: | cut -c 1-2`
> setenv MKL_NUM_THREADS $a
> setenv OMP_NUM_THREADS $a
>
> or
> set a=`grep -e $HOSTNAME $PBS_NODEFILE | wc -l`
>
> My opinion is that this should be set at the OS level with
> MKL_NUM_THREADS etc set by msub/qsub or similar. I've seen problems
> due to other processes (not mine) using more threads than they should.
>
> On Tue, Mar 15, 2011 at 10:35 AM, Peter Blaha
> <pblaha at theochem.tuwien.ac.at> wrote:
>> Setting OMP_NUM_THREAD lets you select to use multithreading or not.
>>
>> A slight modification of our example script on the WIEN2k faq page
>> can even accounts for this and automatically span on eg. 8 core only
>> 4 mpi processes (when OMP_NUM_THREAD=2).
>>
>>
>> Am 15.03.2011 15:31, schrieb Laurence Marks:
>>>
>>> This is an informational report of an issue that can arise with Wien2k
>>> leading to poor performance. I came across it when experimenting with
>>> the "procs=XX" option in msub/qsub with XX=64 as an alternative to the
>>> more standard "nodes=X:ppn=Y".
>>>
>>> In an scf cycle lapwdm and (with the next release) mixer always use
>>> lapack routines (and maybe some others) which by default with mkl are
>>> multithreaded and use multiple cores. For a mpi run with lapw0:
>>> specified lapw0, lapw1 and lapw2 all use mpi. A issue can arise with
>>> msub/qsub (and perhaps other) if the number of cores mkl uses is
>>> larger than what has been allocated and there are other mpi tasks
>>> running on the same node. Then the performance of lapwdm can be bad;
>>> in one case rather than taking 30 seconds it took 14 minutes.
>>>
>>> I am not sure if this is something "fixable" in WIen2k, and there are
>>> ways in qsub/msub to avoid it using nodes=X:ppn=Y. However, procs=XX
>>> is "nice" as Wien2k in mpi does not really need factors of 2
>>> cores/node (I've tested this) and can fit in the gaps.
>>>
>>> N.B., running without mpi and using nodes=X:ppn=1 could also run into
>>> the same problem.
>>>
>>
>> --
>>
>>                                      P.Blaha
>> --------------------------------------------------------------------------
>> Peter BLAHA, Inst.f. Materials Chemistry, TU Vienna, A-1060 Vienna
>> Phone: +43-1-58801-15671             FAX: +43-1-58801-15698
>> Email: blaha at theochem.tuwien.ac.at    WWW:
>> http://info.tuwien.ac.at/theochem/
>> --------------------------------------------------------------------------
>> _______________________________________________
>> Wien mailing list
>> Wien at zeus.theochem.tuwien.ac.at
>> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
>>
>
>
>
> --
> Laurence Marks
> Department of Materials Science and Engineering
> MSE Rm 2036 Cook Hall
> 2220 N Campus Drive
> Northwestern University
> Evanston, IL 60208, USA
> Tel: (847) 491-3996 Fax: (847) 491-7820
> email: L-marks at northwestern dot edu
> Web: www.numis.northwestern.edu
> Chair, Commission on Electron Crystallography of IUCR
> www.numis.northwestern.edu/
> Electron crystallography is the branch of science that uses electron
> scattering and imaging to study the structure of matter.
>



-- 
Laurence Marks
Department of Materials Science and Engineering
MSE Rm 2036 Cook Hall
2220 N Campus Drive
Northwestern University
Evanston, IL 60208, USA
Tel: (847) 491-3996 Fax: (847) 491-7820
email: L-marks at northwestern dot edu
Web: www.numis.northwestern.edu
Chair, Commission on Electron Crystallography of IUCR
www.numis.northwestern.edu/
Electron crystallography is the branch of science that uses electron
scattering and imaging to study the structure of matter.