[Wien] Parallel calculations in cluster - choices

Mon Aug 8 04:38:50 CEST 2011

Dear Professor Marks,
Thank you very much for your helpful and kind reply.
I still need more helps if possible:
1. Can you please send me a copy of your set of commands you mentioned in your last responded email?
2. The system/Node CPU is Xeon X5560 2.80GHz 4cores x 2sockets - so I would go for the benchmarks for bi-Xeon 5320 (overcl 2.67GHz) in the benchmark page - would this means, the most efficient way to run Wien2k in this cluster is with 1 job with 4 threads or 1 job with 8 threads? (this means, I would set in my .bashrc file (export OMP_NUM_THREADS=4 or 8?) - sorry I am not fully understand the benchmark thing. So in this case do I still use 8 kpoint parallel with 8 mpi?
3. You mentioned in your email that there is a way to check the number of k-points for a 1x1x1 cell - would you mind to explain it a bit more or, provide me a formula is any?

Sorry for spending you so much of time for simple questions like these.

Thank you again.

Kind regards,
Qiwen

------Original Message------
From:"Laurence Marks"<L-marks at northwestern.edu>
To:"A Mailing list for WIEN2k users"<wien at zeus.theochem.tuwien.ac.at>
Cc:
Subject:Re: [Wien] Parallel calculations in cluster - choices
Date:08/07/2011 11:07:51 AM(-0500)
>On Sun, Aug 7, 2011 at 5:00 AM, Dr Qiwen  YAO <Yao.Qiwen at nims.go.jp> wrote:
>> Dear Wien2k users,
>> I was running a supercell 3x3x1 calculation for a 4-atom double peroskite compound with spin polarized calculation.
>> The job was killed by the cluster because of wtime limited - see below the relevant error message:
>
>> =>> PBS: job killed: walltime 86419 exceeded limit 86400
>> ----------
>> Question 1.
>> In a case like this - what is the best way for me to continue on the previous calculation (or if it is possible just to re-run the same job again pretending nothing happened - since the calculation itself does not crush) - if I am to restart the job again would Wien2k be able to pick up the best point and continues on? - I could not find anything similar to this in the email achieve.
>
>Yes, just add "-NI" to the runXYZ command (XYZ as appropriate).
>
>>
>> Question 2. The choices of the mpi and k-parallel  - for my future calculation choices.
>> The cluster I am running wien2k with is a 512 node cluster with PBS. Each node is with 8 cores and it is 2.85GB/core in memory/core ratio.
>
>You need to benchmark the speed of your system first. Use the
>benchmark from the Wien2k web page, and try different configurations.
>Work out as well the speed of lapw1 (the slowist program) for
>different sizes for you cluster (and different numbers of nodes).
>
>Note, as well, that in general you want to use a comparable density of
>k-points in reciprocal space. Hence for a 3x3x1 cell you need 1/9 the
>number of k-points that you need for a 1x1x1 cell. In addition, for an
>insulator in general you need less k-points than for a metal. Check
>the number that you need for a 1x1x1 cell then use the same density
>for the 3x3x1 (not the same number).
>
>Last, you should see what is being produced by the script you have,
>i.e. look at the .machines file (cat .machines). I have a small set of
>commands that I have used to control what one gets in a more flexible
>fashion. I will send it to Peter and ask that he puts it on the
>unsupported software page. (I can send it seperately on request, but
>not via the list.)
>
>> I was running 8 k-point parallel and 8 mpi parallel (suggested by the system engineer as I don't have much idea what is the best choice) with these options:
>>
>> #!/usr/bin/tcsh
>> #QSUB2 queue qaM
>> #QSUB2 core 8
>> #QSUB2 mpi 8
>> #QSUB2 smp 1
>>
>> These lines provided by the system support and it seems we do not use dynamic script to find the available nodes/cores in run time but we specify the number of cores/nodes we want to use in our script (we are using a point system to run our scripts) then we submit our job in the queue. I could specify up to maybe 24 nodes x 8 cores/node (#QSUB2 core 192, and  #QSUB2 mpi 192) in the script - would that help speed up the calculation if I did in the current system?
>>
>> The system engineer thought it might not be better for win2k as wien2k is most efficient in k-point parallel. So he suggested me to use 8 k-pont parallel and 8 mpi parallel - even though he is not sure what the best choice is - and I have less idea about the best choice in the current system.
>>
>> Below is the complete script for submitting my previous job:
>> -------------------------
>> #!/usr/bin/tcsh
>> #QSUB2 queue qaM
>> #QSUB2 core 8
>> #QSUB2 mpi 8
>> #QSUB2 smp 1
>>
>> cd ${PBS_O_WORKDIR}
>>
>> source /etc/profile.d/modules.csh
>> module load intel11.1/sgimpt
>>
>> cat $PBS_NODEFILE > .machines_current
>> set aa=`wc .machines_current`
>> echo '#' > .machines
>>
>> # example for an MPI parallel lapw0
>> echo -n 'lapw0:' >> .machines
>> set i=1
>> while ($i < $aa[1])
>> echo -n `cat $PBS_NODEFILE |head -$i | tail -1` ' ' >> .machines
>> @ i ++
>> end
>> echo  `cat $PBS_NODEFILE |head -$i|tail -1` ' ' >> .machines
>>
>> #example for k-point parallel lapw1/2
>> set i=1
>> while ($i <= $aa[1])
>> echo -n '1:' >> .machines
>> head -$i .machines_current |tail -1 >> .machines
>> @ i ++
>> end
>> echo 'granularity:1' >> .machines
>> echo 'extrafine:1' >> .machines
>>
>> runsp_lapw -ec 0.0001 -p
>> ------------------------------
>>
>> Any comment or suggestion would be highly appreciated.
>>
>> Thank you very much.
>> Qiwen
>>
>> ------Original Message------
>> From:"Aaron"<nkleof at gmail.com>
>> To:<Wien at zeus.theochem.tuwien.ac.at>
>> Cc:
>> Subject:[Wien] problem on electronic structure in slab
>> Date:07/29/2011 04:25:23 PM(+0800)
>>>_______________________________________________
>>>Wien mailing list
>>>Wien at zeus.theochem.tuwien.ac.at
>>>http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
>>>
>>> << 1.2.html >>
>>
>> **********************************************************
>>
>> Dr QiWen YAO
>>
>> JSPS Fellow
>> Multifunctional Materials Group
>> Optical and Electronic Materials Unit
>> Environment and Energy Materials Research Division
>>
>> National Institute for Materials Science
>>
>> 1-2-1 Sengen, Tsukuba, Ibaraki 305-0047, Japan
>> Phone: +81-29-851-3354, ext. no. 6482, Fax: +81-29-859-2501
>>
>> **********************************************************
>>
>> _______________________________________________
>> Wien mailing list
>> Wien at zeus.theochem.tuwien.ac.at
>> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
>>
>
>
>
>-- 
>Laurence Marks
>Department of Materials Science and Engineering
>MSE Rm 2036 Cook Hall
>2220 N Campus Drive
>Northwestern University
>Evanston, IL 60208, USA
>Tel: (847) 491-3996 Fax: (847) 491-7820
>email: L-marks at northwestern dot edu
>Web: www.numis.northwestern.edu
>Chair, Commission on Electron Crystallography of IUCR
>www.numis.northwestern.edu/
>Research is to see what everybody else has seen, and to think what
>nobody else has thought
>Albert Szent-Gyorgi
>_______________________________________________
>Wien mailing list
>Wien at zeus.theochem.tuwien.ac.at
>http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien

**********************************************************

Dr QiWen YAO

JSPS Fellow
Multifunctional Materials Group
Optical and Electronic Materials Unit
Environment and Energy Materials Research Division

National Institute for Materials Science

1-2-1 Sengen, Tsukuba, Ibaraki 305-0047, Japan
Phone: +81-29-851-3354, ext. no. 6482, Fax: +81-29-859-2501

**********************************************************