<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none"><!--P{margin-top:0;margin-bottom:0;} p
{margin-top:0;
margin-bottom:0}--></style>
</head>
<body dir="ltr" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p>Thanks a lot for your answer. After re-reading the relevant pages in the User Guide, I am still left with some questions. Specifically, I am working with a system containing 96 atoms (as described in the case.struct-file) and 224 inequivalent k points; i.e.
500 kpoints distributed as a 7x8x8 grid (448 total) reduced to 224 kpoints. Running on 4 nodes each with 16 cores, I want each of the 4 nodes to calculate 56 k points (224/4 = 56). Meanwhile, each node should handle 24 atoms (96/4 = 24).</p>
<p><br>
</p>
<p>Part of my confusion stems from your suggestion that I repeat the line "1:g008:4 [...]" a number of times equal to the number of k points I want to run in parallel, and that each repetition should refer to a different node. The reason is that the line in
question already contains the names of all four nodes that were assigned to the job. However, combining your advice with the example on page 86, the lines should read:</p>
<p><br>
</p>
<p><span style="font-family: "Courier New", monospace;">1:g008</span></p>
<p><span style="font-family: "Courier New", monospace;">1:g021</span></p>
<p><span style="font-family: "Courier New", monospace;">1:g025</span></p>
<p><span style="font-family: "Courier New", monospace;">1:g028 # k points distributed over 4 jobs, running on 1 node each</span><br>
</p>
<p><span style="font-family: "Courier New", monospace;">extrafine:1</span></p>
<p><br>
</p>
<p>As for the parallellization over atoms for dstart and lapw0, I understand that the numbers assigned to each individual node should sum up to the number of atoms in the system, like this:</p>
<p><br>
</p>
<p><span style="font-family: "Courier New", monospace;">dstart:g008:24 g021:24 g025:24 g028:24</span><br>
</p>
<p><span style="font-family: "Courier New", monospace;">lapw0</span><span style="font-family: "Courier New", monospace;">:g008:24 g021:24 g025:24 g028:24</span><br>
</p>
<p><br>
</p>
<p>so the final .machines-file would be a combination of the above pieces. Have I understood this correctly, or am I missing the mark? Also, is there any difference between distributing the k points across four jobs (1 for each node), and across 224 jobs (by
repeating each of the 1:gxxx lines 56 times)?</p>
<p><br>
</p>
<p>Best regards</p>
<p>Christian<br>
</p>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>Fra:</b> Wien <wien-bounces@zeus.theochem.tuwien.ac.at> på vegne af Ruh, Thomas <thomas.ruh@tuwien.ac.at><br>
<b>Sendt:</b> 12. oktober 2020 09:29:37<br>
<b>Til:</b> A Mailing list for WIEN2k users<br>
<b>Emne:</b> Re: [Wien] .machines for several nodes</font>
<div> </div>
</div>
<div>
<p>Hi,</p>
<p><br>
</p>
<p>your .machines is wrong.</p>
<p><br>
</p>
<p>The nodes for <em>lapw1 </em>are prefaced not with "lapw1:" but only with "1:".
<em>lapw2 </em>needs no line, as it takes the same nodes as lapw1 before.</p>
<p><br>
</p>
<p>So an example for your usecase would be:</p>
<p><br>
</p>
<span style="font-family:"Courier New",monospace">#</span>
<p><span style="font-family:"Courier New",monospace">dstart:g008:4 g021:4 g025:4 g028:4</span></p>
<p><span style="font-family:"Courier New",monospace">lapw0:g008:4 g021:4 g025:4 g028:4</span><br>
</p>
<p><span style="font-family:"Courier New",monospace">1:g008:4 g021:4 g025:4 g028:4</span></p>
<p><span style="font-family:"Courier New",monospace">granularity:1</span></p>
<p><span style="font-family:"Courier New",monospace">extrafine:1<br>
</span></p>
<p><br>
</p>
<p>The line starting with "1:" has to be repeated (with different nodes, of course) x times, if you want to run x k-points in parallel (you can find more details about this in the usersguide, pages 84-91).<span style="font-family:"Courier New",monospace"></span></p>
<p><br>
</p>
<p>Regards,</p>
<p>Thomas<br>
</p>
<p><br>
</p>
<p>PS: As a sidenote: Both <em>dstart </em>and <em>lapw0 </em>parallelize over atoms, so 16 nodes might not be the best choice for your example.<br>
</p>
<div dir="ltr" style="color:rgb(33,33,33)">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>Von:</b> Wien <wien-bounces@zeus.theochem.tuwien.ac.at> im Auftrag von Christian Søndergaard Pedersen <chrsop@dtu.dk><br>
<b>Gesendet:</b> Montag, 12. Oktober 2020 09:06<br>
<b>An:</b> wien@zeus.theochem.tuwien.ac.at<br>
<b>Betreff:</b> [Wien] .machines for several nodes</font>
<div> </div>
</div>
<div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
<p>Hello everybody</p>
<p><br>
</p>
<p>I am new to WIEN2k, and am struggling with parallellizing calculations on our HPC cluster beyond what can be achieved using OMP. In particular, I want to execute run_lapw and/or runsp_lapw running on four identical nodes (16 cores each), parallellizing over
k points (unless there's a more efficient scheme). To achieve this, I try to mimic the example from the User Guide (without the extra Alpha node), but my .machines-file does not work the way I intended. This is what I have:</p>
<p><br>
</p>
<p><span style="font-family:"Courier New",monospace">#</span></p>
<p><span style="font-family:"Courier New",monospace">dstart:g008:4 g021:4 g025:4 g028:4</span></p>
<p><span style="font-family:"Courier New",monospace">lapw0:g008:4 g021:4 g025:4 g028:4</span><br>
</p>
<p><span style="font-family:"Courier New",monospace">lapw1:g008:4 g021:4 g025:4 g028:4</span></p>
<p><span style="font-family:"Courier New",monospace">lapw2:g008:4 g021:4 g025:4 g028:4</span><br>
</p>
<p><span style="font-family:"Courier New",monospace">granularity:1</span></p>
<p><span style="font-family:"Courier New",monospace">extrafine:1</span><br>
</p>
<p><br>
</p>
<p>The node names gxxx are read from SLURM_JOB_NODELIST in the submit script, and a couple of regular expressions generate the above lines. Afterwards, my job script does the following:<br>
</p>
<p><br>
</p>
<p><span style="font-family:"Courier New",monospace">srun hostname -s > slurm.hosts</span><br>
<span style="font-family:"Courier New",monospace">run_lapw -p</span><br>
<br>
which results in a job that idles for the entire walltime and finishes with a CPU efficiency of 0.00%. I would appreciate any help in figuring out where I've gone wrong.<br>
</p>
<p><br>
</p>
<p>Best regards<br>
Christian<br>
</p>
</div>
</div>
</div>
</div>
</body>
</html>