<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">From the backtrace, it does look like
it crashed in libmpi.so.1, which I believe is an Open MPI
library. I don't know if it will solve the problem or not, but I
would try a different Open MPI version or recompile Open MPI
(while tweaking the configuration options [
<a class="moz-txt-link-freetext" href="https://software.intel.com/en-us/articles/performance-tools-for-software-developers-building-open-mpi-with-the-intel-compilers">https://software.intel.com/en-us/articles/performance-tools-for-software-developers-building-open-mpi-with-the-intel-compilers</a>
]).<br>
<br>
composer_xe_2015.3.187 => ifort version 15.0.3 [
<a class="moz-txt-link-freetext" href="https://software.intel.com/en-us/articles/intel-compiler-and-composer-update-version-numbers-to-compiler-version-number-mapping">https://software.intel.com/en-us/articles/intel-compiler-and-composer-update-version-numbers-to-compiler-version-number-mapping</a>
]<br>
<br>
In the post at the following link on the Intel forum it looks like
openmpi-1.10.1rc2 (or newer) was recommended for ifort 15.0 (or
newer) to resolve a Fortran run-time library (RTL) issue:<br>
<br>
<a class="moz-txt-link-freetext" href="https://software.intel.com/en-us/forums/intel-fortran-compiler-for-linux-and-mac-os-x/topic/564266">https://software.intel.com/en-us/forums/intel-fortran-compiler-for-linux-and-mac-os-x/topic/564266</a><br>
<br>
On 1/10/2016 3:42 PM, Hu, Wenhao wrote:<br>
</div>
<blockquote
cite="mid:D1716C50-1C58-43A5-A2B1-187ED27EC1CF@uiowa.edu"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<div class=""><br class="">
</div>
<div class="">(I accidentally replied with a wrong title. To
ensure consistency, I send this post again. Maybe the mail list
manager can delete the wrong post for me^)</div>
<div class=""><br class="">
</div>
Hi, Peter:<br class="">
<br class="">
Thank you very much for your reply. By following your suggestion,
I unified the version of all the library to be compiled or
consistent with intel composer xe 2015 (MKL, fftw, openmpi etc.)
and recompiled wien2k. The version of my openmpi is 1.6.5.
However, I still get the same problem. Except for the message I
posted earlier, I also have the following backtrace information of
the process:<br class="">
<br class="">
lapw1c_mpi:14596 terminated with signal 11 at PC=2ab4dac4df79
SP=7fff78b8e310. Backtrace:<br class="">
<br class="">
lapw1c_mpi:14597 terminated with signal 11 at PC=2b847d2a1f79
SP=7fff8ef89690. Backtrace:<br class="">
/opt/openmpi-intel-composer_xe_2015.3.187/1.6.5/lib/libmpi.so.1(MPI_Comm_size+0x59)[0x2ab4dac4df79]<br
class="">
/opt/openmpi-intel-composer_xe_2015.3.187/1.6.5/lib/libmpi.so.1(MPI_Comm_size+0x59)[0x2b847d2a1f79]<br
class="">
/Users/wenhhu/wien2k14/lapw1c_mpi(blacs_pinfo_+0x92)[0x49cf02]<br
class="">
/Users/wenhhu/wien2k14/lapw1c_mpi(blacs_pinfo_+0x92)[0x49cf02]<br
class="">
/opt/intel/composer_xe_2015.3.187/mkl/lib/intel64/libmkl_scalapack_lp64.so(sl_init_+0x21)[0x2b8478d2e171]<br
class="">
/opt/intel/composer_xe_2015.3.187/mkl/lib/intel64/libmkl_scalapack_lp64.so(sl_init_+0x21)[0x2ab4d66da171]<br
class="">
/Users/wenhhu/wien2k14/lapw1c_mpi(parallel_mp_init_parallel_+0x63)[0x463cd3]<br
class="">
/Users/wenhhu/wien2k14/lapw1c_mpi(parallel_mp_init_parallel_+0x63)[0x463cd3]<br
class="">
/Users/wenhhu/wien2k14/lapw1c_mpi(gtfnam_+0x22)[0x426372]<br
class="">
/Users/wenhhu/wien2k14/lapw1c_mpi(MAIN__+0x6c)[0x4493dc]<br
class="">
/Users/wenhhu/wien2k14/lapw1c_mpi(main+0x2e)[0x40d19e]<br class="">
/Users/wenhhu/wien2k14/lapw1c_mpi(gtfnam_+0x22)[0x426372]<br
class="">
/Users/wenhhu/wien2k14/lapw1c_mpi(MAIN__+0x6c)[0x4493dc]<br
class="">
/Users/wenhhu/wien2k14/lapw1c_mpi(main+0x2e)[0x40d19e]<br class="">
/lib64/libc.so.6(__libc_start_main+0xfd)[0x339101ed5d]<br class="">
/Users/wenhhu/wien2k14/lapw1c_mpi[0x40d0a9]<br class="">
/lib64/libc.so.6(__libc_start_main+0xfd)[0x339101ed5d]<br class="">
/Users/wenhhu/wien2k14/lapw1c_mpi[0x40d0a9]<br class="">
<br class="">
Do you think it’s still the problem of my MKL or there’re some
other issues I miss?<br class="">
<br class="">
Best,<br class="">
Wenhao<br class="">
<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">a) Clearly, for a nanowire
simulation the mpi-parallelization is best. Unfortunately, on
some clusters mpi is not set-up properly, or users do not use
the proper mkl-libraries for hthe particular mpi. Please use the
Intel link-library advisor, as was mentioned in previous posts.
The mkl-scalapack will NOT work unless you use proper version of
the blacs_lp64 library.<br class="">
b) As a short term solution you should:<br class="">
<br class="">
i) Use a parallelization with OMP_NUM_THREAD=2. This speeds up
the calculation by nearly a factor of 2 and uses 2 cores in a
single lapw1 without memory increase. ii) Reduce the number of
k-points. I'm pretty sure you can reduce it to 2-4 for scf and
structure optimization. This will save memory due to fewer
k-parallel jobs. iii) During structure optimization you will end
up with very small Si-H and C-H distances. So I'd reduce the H
sphere right now to about 0.6, but keep Si and C large (for C
use around 1.2). With such a setup, a preliminary structure
optimization can be done with RKMAX=2.0, which should later be
checked with 2.5 and 3.0 iv) Use iterative diagonalization !
After the first cycle, this will speed-up the scf by a factor of
5 !! v) And of course, reconsider the size of your "vacuum",
i.e. the seperation of your wires. "Vacuum" is VERY expensive in
terms of memory and one should not set it too large without
test. Optimize your wire with small a,b; then increase the
vacuum later on (x supercell) and check if forces appear again
and distances, ban structure, ... change.<br class="">
<br class="">
<blockquote type="cite" class="">Am 09.01.2016 um 22:07 schrieb
Hu, Wenhao:<br class="">
<br class="">
Hi, Marks and Peter:<br class="">
<br class="">
Thank you for your suggestions. About your reply, I have
several<br class="">
follow-up questions. Actually, I’m using a intermediate
cluster in my<br class="">
university, which has 16 cores and 64 GB memory on standard
nodes. The<br class="">
calculation I’m doing is k-point but not MPI parallelized.
From the :RKM<br class="">
flag I posted in my first email, I estimate that the matrix
size I need<br class="">
for a Rkmax=5+ will be at least 40000. In my current
calculation, the<br class="">
lapw1 program will occupy as large as 3GB on each slot (1 k
point/slot).<br class="">
So I estimate the memory for each slot will be at least 12 GB.
I have 8<br class="">
k points so that 96 GB memory will be required at least (if my<br
class="">
estimation is correct). Considering the current computation
resources I<br class="">
have, this is way too memory demanding. On our clusters,
there’s a 4 GB<br class="">
memory limit for each slot on standard node. Although I can
submit<br class="">
request for high memory node, but their usages are very
competitive<br class="">
among cluster users. Do you have any suggestions on
accomplishing this<br class="">
calculation within the limitation of my cluster?<br class="">
<br class="">
About the details of my calculation, the material I'm looking
at is a<br class="">
hydrogen terminated silicon carbide with 56 atoms. A 1x1x14
k-mesh is<br class="">
picked for k-point sampling. The radius of 1.2 is achieved
from<br class="">
setrmt_lapw actually. Indeed, the radius of hydrogen is too
large and<br class="">
I’m adjusting its radius during the progress of optimization
all the<br class="">
time. The reason why I have such a huge matrix is mainly due
to size of<br class="">
my unit cell. I’m using large unit cell to isolate the
coupling between<br class="">
neighboring nanowire.<br class="">
<br class="">
Except for the above questions, I also met some problems in
mpi<br class="">
calculation. By following Marks’ suggestion on parallel
calculation, I<br class="">
want to test the efficiency of mpi calculation since I only
used k-point<br class="">
parallelized calculation before. The MPI installed on my
cluster is<br class="">
openmpi. In the output file, I get the following error:<br
class="">
<br class="">
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br
class="">
LAPW0 END<br class="">
<br class="">
lapw1c_mpi:19058 terminated with signal 11 at PC=2b56d9118f79<br
class="">
SP=7fffc23d6890. Backtrace:<br class="">
...<br class="">
mpirun has exited due to process rank 14 with PID 19061 on<br
class="">
node neon-compute-2-25.local exiting improperly. There are two
reasons<br class="">
this could occur:<br class="">
<br class="">
1. this process did not call "init" before exiting, but others
in<br class="">
the job did. This can cause a job to hang indefinitely while
it waits<br class="">
for all processes to call "init". By rule, if one process
calls "init",<br class="">
then ALL processes must call "init" prior to termination.<br
class="">
<br class="">
2. this process called "init", but exited without calling
"finalize".<br class="">
By rule, all processes that call "init" MUST call "finalize"
prior to<br class="">
exiting or it will be considered an "abnormal termination"<br
class="">
<br class="">
This may have caused other processes in the application to be<br
class="">
terminated by signals sent by mpirun (as reported here).<br
class="">
--------------------------------------------------------------------------<br
class="">
Uni_+6%.scf1up_1: No such file or directory.<br class="">
grep: *scf1up*: No such file or directory<br class="">
<br class="">
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br
class="">
<br class="">
The job script I’m using is:<br class="">
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br
class="">
!/bin/csh -f<br class="">
# -S /bin/sh<br class="">
#<br class="">
#$ -N uni_6<br class="">
#$ -q MF<br class="">
#$ -m be<br class="">
#$ -M wenhao...@<a moz-do-not-send="true"
href="http://uiowa.edu" class="">uiowa.edu</a> <<br
class="">
<a class="moz-txt-link-freetext" href="mailto:wenhao...@">mailto:wenhao...@</a><a moz-do-not-send="true"
href="http://uiowa.edu" class="">uiowa.edu</a><br class="">
<blockquote type="cite" class=""><br class="">
</blockquote>
#$ -pe smp 16<br class="">
#$ -cwd<br class="">
#$ -j y<br class="">
<br class="">
cp $PE_HOSTFILE hostfile<br class="">
echo "PE_HOSTFILE:"<br class="">
echo $PE_HOSTFILE<br class="">
rm .machines<br class="">
echo granularity:1 >>.machines<br class="">
while read hostname slot useless; do<br class="">
i=0<br class="">
l0=$hostname<br class="">
while [ $i -lt $slot ]; do<br class="">
echo 1:$hostname:2 >>.machines<br class="">
let i=i+2<br class="">
done<br class="">
done<hostfile<br class="">
<br class="">
echo lapw0:$l0:16 >>.machines<br class="">
<br class="">
runsp_lapw -p -min -ec 0.0001 -cc 0.001 -fc 0.5<br class="">
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br
class="">
<br class="">
Is there any mistake I made or something missing in my script?<br
class="">
<br class="">
Thank your very much for your help.<br class="">
<br class="">
Wenhao<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">I do not know many compounds,
for which an RMT=1.2 bohr for H makes<br class="">
any sense (maybe LiH). Use setrmt and follow the suggestion.
Usually,<br class="">
H spheres of CH or OH bonds should be less than 0.6 bohr.<br
class="">
Experimental H-position are often very unreliable.<br
class="">
How many k-points ? Often 1 k-point is enough for 50+ atoms
(at least<br class="">
at the beginning), in particular when you ahve an insulator.<br
class="">
Otherwise, follow the suggestions of L.Marks about
parallelization.<br class="">
<br class="">
<br class="">
Am 08.01.2016 um 07:28 schrieb Hu, Wenhao:<br class="">
<br class="">
Hi, all:<br class="">
<br class="">
I have some confusions on the Rkm in calculations with 50+
atoms. In<br class="">
my wien2k,<br class="">
the NATMAX and NUME are set to 15000 and 1700. With the
highest NE<br class="">
and NAT, the<br class="">
Rkmax can only be as large as 2.05, which is much lower than
the<br class="">
suggested<br class="">
value in FAQ page of WIEN2K (the smallest atom in my case is
a H atom<br class="">
with<br class="">
radius of 1.2). By checking the :RKM flag in case.scf, I
have the<br class="">
following<br class="">
information:<br class="">
<br class="">
:RKM : MATRIX SIZE 11292LOs: 979 RKM= 2.05 WEIGHT= 1.00
PGR:<br class="">
<br class="">
With such a matrix size, the single cycle can take as long
as two and<br class="">
half<br class="">
hours. Although I can increase the NATMAX and NUME to raise
Rkmax, the<br class="">
calculation will be way slower, which will make the
optimization<br class="">
calculation<br class="">
almost impossible. Before making convergence test on Rkmax,
can<br class="">
anyone tell me<br class="">
whether such a Rkmax is a reasonable value?<br class="">
<br class="">
If any further information is needed, please let me know.
Thanks in<br class="">
advance.<br class="">
<br class="">
Best,<br class="">
Wenhao<br class="">
</blockquote>
_______________________________________________<br class="">
Wien mailing list<br class="">
<a moz-do-not-send="true"
href="mailto:Wien@zeus.theochem.tuwien.ac.at" class="">Wien@zeus.theochem.tuwien.ac.at</a> <<br
class="">
<a moz-do-not-send="true"
href="mailto:Wien@zeus.theochem.tuwien.ac.at" class="">mailto:Wien@zeus.theochem.tuwien.ac.at</a><br
class="">
<blockquote type="cite" class=""><br class="">
</blockquote>
<br class="">
<br class="">
<a moz-do-not-send="true"
href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien"
class="">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a><br
class="">
<br class="">
<br class="">
SEARCH the MAILING-LIST at:<br class="">
<br class="">
<br class="">
<a moz-do-not-send="true"
href="http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html"
class="">http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html</a><br
class="">
</blockquote>
<br class="">
<br class="">
--<br class="">
--------------------------------------------------------------------------<br
class="">
Peter BLAHA, Inst.f. Materials Chemistry, TU Vienna, A-1060
Vienna<br class="">
Phone: +43-1-58801-165300 FAX: +43-1-58801-165982<br
class="">
Email: bl...@<a moz-do-not-send="true"
href="http://theochem.tuwien.ac.at" class="">theochem.tuwien.ac.at</a> <<br
class="">
<a moz-do-not-send="true" href="http://theochem.tuwien.ac.at"
class="">http://theochem.tuwien.ac.at</a><br class="">
<blockquote type="cite" class=""><br class="">
</blockquote>
WIEN2k:<br class="">
<br class="">
<a moz-do-not-send="true" href="http://www.wien2k.at" class="">http://www.wien2k.at</a><br
class="">
<br class="">
<br class="">
WWW:<br class="">
<br class="">
<a moz-do-not-send="true"
href="http://www.imc.tuwien.ac.at/staff/tc_group_e.php"
class="">http://www.imc.tuwien.ac.at/staff/tc_group_e.php</a><br
class="">
</blockquote>
</blockquote>
</body>
</html>