[Wien] 24k points on 36processors ??. (a Fractional k-point per core)

[Ashwani Kumar]

Dear Sir,
         Hyper-threading is disabled (just checked with facility expert).
So 12 physical cores per node (intel xeon nehalem based arch.). Available
Memory 4gb/core (48gb/node).
         Lapw1 stops with error "insufficient virtual memory". So i thought
better to use 36 cores for 24k points as extra (48gb) memory will be
available. I am using pbs queuing system (wien2k  V19.1 compiled with
openmpi_parallelization) which generates *.machine file when jobscript
executed. Then how to set the omp_thread in *.machine file.  (jobscript
file attached for your reference).

thanks,
A. kumar

On Thu, Dec 12, 2019 at 2:55 PM Pavel Ondračka <pavel.ondracka at email.cz>
wrote:

> Hi,
>
> do you have hyperthreading or not (in other words does this number of
> 12 already mean there are 6 physical CPUs and 12 virtuals, or 12
> physical)? This would influence the advice maybe a bit...
>
> Otherwise you need to experiment, the optimal setting is heavily
> dependent on your specific CPU, memory speed and what you are
> calculating (system size).
>
> When talking about the 24 kpoints and 36 processors, than running
> 4kpoints on each node  (12 kpoints in parallel) with 3 openmp threads
> each might be a reasonable setting.
>
> It is also possible that just leaving some cores idle might be the best
> thing to do (as running a lot of k-points in parallel you can get
> limited by the memory speed so leaving some cores idle means more
> memory bandwidth for the others):
> This would correspond to running 8 kpoints on each node or 4 kpoints on
> each node with 2 openmp threads each.
>
> The linux kernel and modern processors are also usually good at
> handling some small overload and load balancing so you can also try to
> overload the system a bit, i.e., 8kpoints per node with 2 openmp
> threads each.
>
> Just try the different settings (single lapw1 run for each should be
> enough to get some idea) and compare the timings.
>
> Best regards
> Pavel
>
> BTW for lapw0 I would go with something like 3 MPI processes per node
> with 4 OpenMP threads for each in this case.
>
> On Thu, 2019-12-12 at 12:28 +0530, Ashwani Kumar wrote:
> > Hi,
> >    This is related to no. of k-points which we provide during the
> > initilization. No. of k-gen points given ; 120 with shifted mesh.
> > Irr. k-points : 24k points. Running job on 3 nodes (12 x3 processors,
> > 48 gb x 3 Ram). Job running on 24 processors only (with granularity:
> > 1, extrafine:1 in *.machine file) which means 1kpoint/1-core. How can
> > 24 k-points be made to run on 36  cores ?. Or how can 24 kpoints can
> > be distributed equally between 36 cores (or let's say 12 kpoints on
> > 24 processors to make calculation converge faster).
> >
> > thanks,
> > A. Kumar
> > _______________________________________________
> > Wien mailing list
> > Wien at zeus.theochem.tuwien.ac.at
> > http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
> > SEARCH the MAILING-LIST at:
> > http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html
>
> _______________________________________________
> Wien mailing list
> Wien at zeus.theochem.tuwien.ac.at
> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
> SEARCH the MAILING-LIST at:
> http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://zeus.theochem.tuwien.ac.at/pipermail/wien/attachments/20191212/b21ed7d9/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: runjob
Type: application/octet-stream
Size: 30 bytes
Desc: not available
URL: <http://zeus.theochem.tuwien.ac.at/pipermail/wien/attachments/20191212/b21ed7d9/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jobkzx3
Type: application/octet-stream
Size: 922 bytes
Desc: not available
URL: <http://zeus.theochem.tuwien.ac.at/pipermail/wien/attachments/20191212/b21ed7d9/attachment-0001.obj>