<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="NL" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-GB">Dear WIEN2k users,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">I have the following problem. I am trying to do parallel computing on a cluster. Whenever I run a job on the cluster on one node both the MPI and k-point parallelization work fine. However, when I try to go to several
nodes the job does not do anything. The script just gets stuck on lapw0 whenever MPI is implemented, the k-point parallelization does not give a problem when running on multiple nodes. Additionally, if instead I try to run a job where lapw0 is running parallel
only on one node on multiple processors, but lapw1 and lapw2 are run on multiple nodes, then again there is no problem.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">I do not get an error while running lapw0 parallel over multiple nodes: the job doesn’t do anything. The assigned nodes are scheduled to the job, but the load on the nodes stays at 0%. When I forcibly stop the job then
I get these following errors:<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">[mpiexec@n05-38] HYDU_sock_write (../../utils/sock/sock.c:417): write error (Bad file descriptor)<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">[mpiexec@n05-38] HYD_pmcd_pmiserv_send_signal (../../pm/pmiserv/pmiserv_cb.c:252): unable to write data to proxy<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">[mpiexec@n05-38] ui_cmd_cb (../../pm/pmiserv/pmiserv_pmci.c:174): unable to send signal downstream<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">[mpiexec@n05-38] HYDT_dmxu_poll_wait_for_event (../../tools/demux/demux_poll.c:76): callback returned error status<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">[mpiexec@n05-38] HYD_pmci_wait_for_completion (../../pm/pmiserv/pmiserv_pmci.c:500): error waiting for event<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">[mpiexec@n05-38] main (../../ui/mpich/mpiexec.c:1130): process manager error waiting for completion<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><br>
In the example above I tried to run a job on nodes n05-32 and n05-38.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">The operating system on the cluster is <a href="http://www.centos.org/" target="_blank"><span style="color:windowtext;text-decoration:none">CentOS 7</span></a>. The cluster consists of a master node where Mau and Torque
are running (PBS implementation). The cluster is set up so that rsh can be used instead of ssh.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">I am using intel’s parallel studio 2016: <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">My .machines file looked like this:<br>
<br>
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">n05-32:10<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">n05-38:10<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">lapw0:n05-32:10 n05-38:10<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">extrafine=1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">granularity=1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">In the script I submit for the jobs I source my .bashrc file:<br>
source /home/.bashrc<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">The .bashrc file has the following lines:<br>
source /opt/ud/intel_xe_2016/parallel_studio_xe_2016.3.067/compilers_and_libraries_2016/linux/mkl/bin/mklvars.sh intel64<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">source /opt/ud/intel_xe_2016/parallel_studio_xe_2016.3.067/compilers_and_libraries_2016/linux/bin/compilervars.sh intel64<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB" style="color:black">source /opt/ud/intel_xe_2016/parallel_studio_xe_2016.3.067/compilers_and_libraries_2016/linux/bin/iccvars.sh intel64<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB" style="color:black">source /opt/ud/intel_xe_2016/parallel_studio_xe_2016.3.067/compilers_and_libraries_2016/linux/mpi/intel64/bin/mpivars.sh intel64<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB" style="color:black"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB" style="color:black">export PATH=$PATH:/opt/ud/intel_xe_2016/parallel_studio_xe_2016.3.067/compilers_and_libraries_2016/linux/mpi/intel64/bin<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB" style="color:black">export PATH=$PATH:/opt/ud/intel_xe_2016/parallel_studio_xe_2016.3.067/compilers_and_libraries_2016/linux/mkl/include<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB" style="color:black">export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/ud/intel_xe_2016/parallel_studio_xe_2016.3.067/compilers_and_libraries_2016/linux/mpi/intel64/lib<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB" style="color:black">export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/ud/intel_xe_2016/parallel_studio_xe_2016.3.067/compilers_and_libraries_2016/linux/mkl/lib/intel64<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB" style="color:black"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB" style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Along with the rest of the alias’ and environment variables that ./userconfig_lapw sets up.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Some more parameters:<br>
System: LI<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">ifort compiler<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">icc c compiler<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">mpiifort compiler<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><br>
Compiler options: -O1 –FR –mp1 –w –prec_div –pc80 –pad –ip –DINTEL_VML –traceback –assume buffered_io –I/<span style="color:black">opt/ud/intel_xe_2016/parallel_studio_xe_2016.3.067/compilers_and_libraries_2016/linux/mkl/include<o:p></o:p></span></span></p>
<p class="MsoPlainText"><span lang="EN-GB" style="color:black"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB" style="color:black">Linker flags: $(FOPT) –L/opt/ud/intel_xe_2016/parallel_studio_xe_2016.3.067/compilers_and_libraries_2016/linux/mkl/lib/intel64 –lpthread<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB" style="color:black"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB" style="color:black">Preprocessors flags: ‘-DParallel’<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB" style="color:black"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB" style="color:black">R_LIBS: -lmkl_lapack95_lp64 –lmkl_intel_lp64 –lmkl_intel_thread –lmkl_core –openmp –lpthread –liomp5<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:black">SCALAPACK: -L/opt/ud/intel_xe_2016/parallel_studio_xe_2016.3.067/compilers_and_libraries_2016/linux/mkl/lib/intel64 –lmkl_scalapack_lp64 -L/opt/ud/intel_xe_2016/parallel_studio_xe_2016.3.067/compilers_and_libraries_2016/linux/mkl/lib/intel64
–lmkl_blacs_intelmpi_lp64<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:black">And my parallel options file looks like this:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span lang="EN-GB" style="color:black">setenv TASKSET "no"<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span lang="EN-GB" style="color:black">if ( ! $?USE_REMOTE ) setenv USE_REMOTE 1<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span lang="EN-GB" style="color:black">if ( ! $?MPI_REMOTE ) setenv MPI_REMOTE 1<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span lang="EN-GB" style="color:black">setenv WIEN_GRANULARITY 1<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span lang="EN-GB" style="color:black">setenv DELAY 0.1<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span lang="EN-GB" style="color:black">setenv SLEEPY 1<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span lang="EN-GB" style="color:black">setenv WIEN_MPIRUN "/opt/ud/intel_xe_2016/parallel_studio_xe_2016.3.067/compilers_and_libraries_2016/linux/mpi/intel64/bin/mpirun -ppn $PBS_NUM_PPN -np _NP_ -machinefile
_HOSTS_ _EXEC_"<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB" style="color:black">setenv CORES_PER_NODE 20<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB" style="color:black"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB" style="color:black">I downloaded my fftw package and configured it with the options F77=ifort CC=icc MPICC=mpiicc<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB" style="color:black">Similarly for the LIBXC package: FC=ifort CC=icc<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB" style="color:black"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB" style="color:black">Any help is appreciated.
<br>
Luigi Maduro<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB" style="color:black"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB" style="color:black"><o:p> </o:p></span></p>
</div>
</body>
</html>