<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Verdana
}
--></style>
</head>
<body class='hmmessage'>
Hi Marks,<br><br> In addition to what I have sent in my previous email, I would like to mention that<br>if I use 8 processors instead of 4 processors, I get the segmentation error at lapw0.<br><br>Thanks,<br>Fhokrul <br><br><br><hr id="stopSpelling">From: fislam@hotmail.com<br>To: wien@zeus.theochem.tuwien.ac.at<br>Date: Sat, 30 Jan 2010 18:51:59 +0000<br>Subject: Re: [Wien] Fwd: MPI segmentation fault<br><br>
<style>
.ExternalClass .ecxhmmessage P
{padding:0px;}
.ExternalClass body.ecxhmmessage
{font-size:10pt;font-family:Verdana;}
</style>
<style>
.ExternalClass .ecxhmmessage P
{padding:0px;}
.ExternalClass body.ecxhmmessage
{font-size:10pt;font-family:Verdana;}
</style>
Hi Marks,<br><br> I have followed your suggestions and have used openmpi 1.4.1 compiled with icc.<br>I also have compiled fftw with cc instead of gcc and recompiled Wien2k with mpirun option<br>in parallel_options:<br><br>current:MPIRUN:mpirun -np _NP_ -machinefile _HOSTS_ _EXEC_ -x LD_LIBRARY_PATH<br> <br>Although I didn't get segmentation fault but the job still crashes at lapw1 with a different error <br>message. I have pasted case.dayfile and case.error below along with ompi_info and stacksize<br>info. I am not even sure where to look for the solution. Please let me know if you have any<br>suggestions regarding this MPI problem.<br><br>Thanks,<br>Fhokrul <br><br>case.dayfile:<br><br> cycle 1 (Sat Jan 30 16:49:55 CET 2010) (200/99 to go)<br><br>> lapw0 -p (16:49:55) starting parallel lapw0 at Sat Jan 30 16:49:56 CET 2010<br>-------- .machine0 : 4 processors<br>1863.235u 21.743s 8:21.32 376.0% 0+0k 0+0io 1068pf+0w<br>> lapw1 -c -up -p (16:58:17) starting parallel lapw1 at Sat Jan 30 16:58:18 CET 2010<br>-> starting parallel LAPW1 jobs at Sat Jan 30 16:58:18 CET 2010<br>running LAPW1 in parallel mode (using .machines)<br>1 number_of_parallel_jobs<br> mn117.mpi mn117.mpi mn117.mpi mn117.mpi(1) 1263.782u 28.214s 36:47.58 58.5% 0+0k 0+0io 49300pf+0w<br>** LAPW1 crashed!<br>1266.358u 37.286s 36:53.31 58.8% 0+0k 0+0io 49425pf+0w<br>error: command /disk/global/home/eishfh/Wien2k_09_2/lapw1cpara -up -c uplapw1.def failed<br><br>Error file:<br><br> LAPW0 END<br> LAPW0 END<br> LAPW0 END<br> LAPW0 END<br>--------------------------------------------------------------------------<br>mpirun noticed that process rank 0 with PID 8837 on node mn117.local exited on signal 9 (Killed).<br><br><span style="font-size: 10pt; font-family: 'Arial','sans-serif';"><br>[eishfh@milleotto
s110]$ ompi_info</span>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';"><span style=""> </span>Package: Open MPI
root@milleotto.local Distribution</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';"><span style=""> </span>Open MPI: 1.4.1</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';"><span style=""> </span>Prefix:
/sw/pkg/openmpi/1.4.1/intel/11.1</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';"><span style=""> </span>Configured architecture:
x86_64-unknown-linux-gnu</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';"><span style=""> </span>Configure host: milleotto.local</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';"><span style=""> </span>Configured by: root</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';"><span style=""> </span>Configured on: Sat Jan 16 19:40:36
CET 2010</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';"><span style=""> </span>Configure host: milleotto.local</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';"><span style=""> </span>Built host: milleotto.local</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';">Fortran90 bindings
size: small</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';"><span style=""> </span>C compiler: icc</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';"><span style=""> </span>C compiler absolute:
/sw/pkg/intel/11.1.064//bin/intel64/icc</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';"><span style=""> </span>C++ compiler: icpc</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';"><span style=""> </span>C++ compiler absolute:
/sw/pkg/intel/11.1.064//bin/intel64/icpc</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';"><span style=""> </span>Fortran77 compiler: ifort</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';"><span style=""> </span>Fortran77 compiler abs:
/sw/pkg/intel/11.1.064//bin/intel64/ifort</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';"><span style=""> </span>Fortran90 compiler: ifort</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';"><span style=""> </span>Fortran90 compiler abs:
/sw/pkg/intel/11.1.064//bin/intel64/ifort</span></p>
<br>stacksize:<br><br><style>
.ExternalClass p.ecxMsoNormal, .ExternalClass li.ecxMsoNormal, .ExternalClass div.ecxMsoNormal
{margin-right:0in;margin-bottom:10.0pt;margin-left:0in;line-height:115%;font-size:11.0pt;font-family:'Calibri','sans-serif';}
.ExternalClass .ecxMsoChpDefault
{;}
.ExternalClass .ecxMsoPapDefault
{margin-bottom:10.0pt;line-height:115%;}
@page Section1
{size:8.5in 11.0in;}
.ExternalClass div.ecxSection1
{page:Section1;}
</style>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';"><span style=""> </span>[eishfh@milleotto s110]$ ulimit -a</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';">core file size<span style=""> </span>(blocks, -c) 0</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';">data seg size<span style=""> </span>(kbytes, -d) unlimited</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';">scheduling
priority<span style=""> </span>(-e) 0</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';">file size<span style=""> </span><span style=""> </span>(blocks, -f) unlimited</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';">pending signals<span style=""> </span>(-i) 73728</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';">max locked
memory<span style=""> </span>(kbytes, -l) 32</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';">max memory size<span style=""> </span>(kbytes, -m) unlimited</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';">open files<span style=""> </span>(-n) 1024</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';">pipe size<span style=""> </span>(512 bytes, -p) 8</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';">POSIX message
queues<span style=""> </span>(bytes, -q) 819200</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';">real-time
priority<span style=""> </span>(-r) 0</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';">stack size<span style=""> </span>(kbytes, -s) unlimited</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';">cpu time<span style=""> </span>(seconds, -t) unlimited</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';">max user
processes<span style=""> </span>(-u) 73728</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';">virtual memory<span style=""> </span>(kbytes, -v) unlimited</span></p>
<p class="ecxMsoNormal" style="margin-bottom: 0.0001pt; line-height: normal;"><span style="font-size: 10pt; font-family: 'Arial','sans-serif';">file locks<span style=""> </span>(-x) unlimited</span></p>
<br><br><br><br>> <br>> In essence, you have a mess and you are going to have to talk to your<br>> sysadmin (hikmpn) to get things sorted out. Issues:<br>> <br>> a) You have openmpi-1.3.3. This works for small problems, fails for<br>> large ones. This needs to be updated to 1.4.0 or 1.4.1 (the older<br>> versions of openmpi have bugs).<br>> b) The openmpi was compiled with ifort 10.1 but you are using 11.1.064<br>> for Wien2k -- could lead to problems.<br>> c) The openmpi was compiled with gcc and ifort 10.1, not icc and ifort<br>> which could lead to problems.<br>> d) The fftw library you are using was compiled with gcc not icc, this<br>> could lead to problems.<br>> e) Some of the shared libraries are in your LD_LIBRARY_PATH, you will<br>> need to add -x LD_LIBRARY_PATH to how mpirun is called (in<br>> $WIENROOT/parallel_options) -- look at man mpirun.<br>> f) I still don't know what the stack limits are on your machine --<br>> this can lead to severe problems in lapw0_mpi<br><br> <br><hr>Hotmail: Trusted email with Microsoft’s powerful SPAM protection. <a href="https://signup.live.com/signup.aspx?id=60969">Sign up now.</a> <br /><hr />Hotmail: Free, trusted and rich email service. <a href='https://signup.live.com/signup.aspx?id=60969' target='_new'>Get it now.</a></body>
</html>