[Wien] MPI segmentation fault

Fri Jan 29 15:20:08 CET 2010

A brief explanation of why I was requesting all this information:

1) ompi_info will tell what the openmpi version is, and whether it was
correctly compiled with icc/ifort or not. I know that versions 1.3.2
and 1.3.3 have problems.
2) Information about what LD_LIBRARY_PATH, PATH and an ldd on
lapw0_mpi will say whether your sysadmin has setup shared libraries in
ld.so.conf so they are global or not. You may need to add "-x
LD_LIBRARY_PATH -x PATH " to MPIRUN as by default openmpi does not
export environmental variables.
3) Information about what the stacksize parameter shows without it
being set from within a bash/csh/tcsh will say whether your sysadmin
has set the soft limit high or not. Unfortunately the way openmpi
works this is not passed to other processes from the parent and some
fixes are needed if the soft limit is low.
4) Information about what process is crashing is basic, is it a bug in
lapw2_mpi (postded a few weeks ago), or elsewhere?

On Thu, Jan 28, 2010 at 8:04 AM, Laurence Marks
<L-marks at northwestern.edu> wrote:
> Also, in 3) while still on the node do
> which lapw0_mpi
> which mpirun
>
> And include this information.
>
> On Thu, Jan 28, 2010 at 8:01 AM, Laurence Marks
> <L-marks at northwestern.edu> wrote:
>> We need a bit more information than this.
>>
>> 1) Please do "ompi_info " and paste the output to the end of your
>> response to this email.
>>
>> 2) Also paste the output of "echo $LD_LIBRARY_PATH"
>>
>> 3) If you have in your .bashrc a "ulimit -s unlimited" please edit
>> this (temporarily) out, then ssh into one of the child nodes. (If you
>> are using csh/tcsh edit out "limit stacksize unlimited). Then do a
>> "ulimit -s" or in csh/tcsh "limit stacksize", and include the result.
>> If you are using qsub you may need to launch an interactive job. Also
>> do "echo $LD_LIBRARY_PATH" while on the node, pasting the result, and
>> "ldd $WIENROOT/lapw0_mpi". (If $LD_LIBRARY_PATH is not blank, repeat
>> the "ldd $WIENROOT/lapw0_mpi" after doing a LD_LIBRARY_PATH= " ", or
>> setenv LD_LIBRARY_PATH .)
>>
>> 4) Tell us which program gave a SIGSEV, lapw0_mpi. lapw1_mpi or lapw2_mpi?
>>
>> 2010/1/28 Md. Fhokrul Islam <fislam at hotmail.com>:
>>> Hi Marks,
>>>
>>>     Thank you very much for your reply. I am using Wien2k_09.2 version and I
>>> have used the following
>>> OPTIONS file for MPI compilation. I would like to mention that MPI works
>>> fine when I tested with an 8 atom
>>> system.
>>>
>>>
>>> current:FOPT:-FR -mp1 -w -prec_div -pc80 -pad -align -DINTEL_VML -traceback
>>> current:FPOPT:$(FOPT)
>>> current:LDFLAGS:-L/sw/pkg/mkl/10.0/lib/em64t/
>>> current:DPARALLEL:'-DParallel'
>>> current:R_LIBS:-lmkl_intel_lp64 -lmkl_sequential -lmkl_core -liomp5 -lsvml
>>> -lpthread
>>> current:RP_LIBS:-L/sw/pkg/mkl/10.0/lib/em64t/ -lmkl_scalapack_lp64
>>> -lmkl_blacs_openmpi_lp64 -L/home/eishfh/fftw-2.1.5-gcc/lib -lfftw_mpi -lfftw
>>> -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -liomp5 -lsvml -lpthread
>>> current:MPIRUN:mpirun -np _NP_ -machinefile _HOSTS_ _EXEC_
>>>
>>>
>>> Thanks,
>>> Fhokrul
>>>
>>>
>>>
>>>> Date: Thu, 28 Jan 2010 07:16:37 -0600
>>>> From: L-marks at northwestern.edu
>>>> To: wien at zeus.theochem.tuwien.ac.at
>>>> Subject: Re: [Wien] MPI segmentation fault
>>>>
>>>> What version of mpi are you using -- please be specific including the
>>>> release.
>>>>
>>>> 2010/1/28 Md. Fhokrul Islam <fislam at hotmail.com>:
>>>> > Dear Wien2k users,
>>>> >
>>>> >     I am trying to do a surface supercell calculation with 96 atoms (1
>>>> > k-point) using MPI. I have
>>>> > used 8 processors for this job but it crashes in 1st cycle with an error
>>>> > message:
>>>> >
>>>> > mpirun noticed that process rank 7 with PID 6532 on node mn003.mpi
>>>> > exited on
>>>> > signal 11 (Segmentation fault).
>>>> >
>>>> >    Since many of you have experience in running large systems with MPI,
>>>> > I am
>>>> > wondering if
>>>> > anyone can suggest me how to fix this problem.
>>>> >
>>>> >
>>>> > Thanks,
>>>> > Fhokrul
>>>> >
>>>> > ________________________________
>>>> > Hotmail: Powerful Free email with security by Microsoft. Get it now.
>>>> > _______________________________________________
>>>> > Wien mailing list
>>>> > Wien at zeus.theochem.tuwien.ac.at
>>>> > http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
>>>> >
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Laurence Marks
>>>> Department of Materials Science and Engineering
>>>> MSE Rm 2036 Cook Hall
>>>> 2220 N Campus Drive
>>>> Northwestern University
>>>> Evanston, IL 60208, USA
>>>> Tel: (847) 491-3996 Fax: (847) 491-7820
>>>> email: L-marks at northwestern dot edu
>>>> Web: www.numis.northwestern.edu
>>>> Chair, Commission on Electron Crystallography of IUCR
>>>> www.numis.northwestern.edu/
>>>> Electron crystallography is the branch of science that uses electron
>>>> scattering and imaging to study the structure of matter.
>>>> _______________________________________________
>>>> Wien mailing list
>>>> Wien at zeus.theochem.tuwien.ac.at
>>>> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
>>>
>>> ________________________________
>>> Hotmail: Trusted email with powerful SPAM protection. Sign up now.
>>> _______________________________________________
>>> Wien mailing list
>>> Wien at zeus.theochem.tuwien.ac.at
>>> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
>>>
>>>
>>
>>
>>
>> --
>> Laurence Marks
>> Department of Materials Science and Engineering
>> MSE Rm 2036 Cook Hall
>> 2220 N Campus Drive
>> Northwestern University
>> Evanston, IL 60208, USA
>> Tel: (847) 491-3996 Fax: (847) 491-7820
>> email: L-marks at northwestern dot edu
>> Web: www.numis.northwestern.edu
>> Chair, Commission on Electron Crystallography of IUCR
>> www.numis.northwestern.edu/
>> Electron crystallography is the branch of science that uses electron
>> scattering and imaging to study the structure of matter.
>>
>
>
>
> --
> Laurence Marks
> Department of Materials Science and Engineering
> MSE Rm 2036 Cook Hall
> 2220 N Campus Drive
> Northwestern University
> Evanston, IL 60208, USA
> Tel: (847) 491-3996 Fax: (847) 491-7820
> email: L-marks at northwestern dot edu
> Web: www.numis.northwestern.edu
> Chair, Commission on Electron Crystallography of IUCR
> www.numis.northwestern.edu/
> Electron crystallography is the branch of science that uses electron
> scattering and imaging to study the structure of matter.
>

-- 
Laurence Marks
Department of Materials Science and Engineering
MSE Rm 2036 Cook Hall
2220 N Campus Drive
Northwestern University
Evanston, IL 60208, USA
Tel: (847) 491-3996 Fax: (847) 491-7820
email: L-marks at northwestern dot edu
Web: www.numis.northwestern.edu
Chair, Commission on Electron Crystallography of IUCR
www.numis.northwestern.edu/
Electron crystallography is the branch of science that uses electron
scattering and imaging to study the structure of matter.