<html><body><div>I'm putting this also back to the list after I received several private emails.</div><div><br></div><div>Your timing and the ldd shows that you are linking against reference lapack and blas. You need to replace -llapack -lblas in R_LIBS with -lopenblas (this was discussed before in this thread: https://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/msg18194.html )</div><div><br></div><div>Also your config is a weird mix of ifort and gfortran options, which results in a ton of errors for the parallel programs (as was shown in another off-the-list email). At this moment this doesn't matter as we need to make the serial stuff working first.</div><div><br></div><div>Best regards<br></div><div>Pavel<br></div><blockquote data-email="indranil.mal@gmail.com"><div dir="ltr"><br>grep "TIME HAMILT" test_case.output1<br> TIME HAMILT (CPU) = 22.8, HNS = 12.3, HORB = 0.0, DIAG = 78.9<br> TIME HAMILT (WALL) = 22.9, HNS = 12.4, HORB = 0.0, DIAG = 78.9<br></div></blockquote><p> <br></p><blockquote data-email="indranil.mal@gmail.com"><div dir="ltr">current:FOPT:-ffree-form -O2 -ffree-line-length-none<br>current:FPOPT:-O1 -FR -mp1 -w -prec_div -pc80 -pad -ip -DINTEL_VML -traceback -assume buffered_io -I$(MKLROOT)/include<br>current:LDFLAGS:$(FOPT)<br>current:DPARALLEL:'-DParallel'<br>current:R_LIBS:-llapack -lblas -lpthread<br>current:FFTWROOT:<br>current:FFTW_VERSION:<br>current:FFTW_LIB:<br>current:FFTW_LIBNAME:<br>current:LIBXCROOT:/opt/etsf/<br>current:LIBXC_FORTRAN:xcf03<br>current:LIBXC_LIBNAME:xc<br>current:LIBXC_LIBDNAME:lib/<br>current:SCALAPACKROOT:<br>current:SCALAPACK_LIBNAME:<br>current:BLACSROOT:<br>current:BLACS_LIBNAME:<br>current:ELPAROOT:<br>current:ELPA_VERSION:<br>current:MPIRUN:mpirun -np _NP_ -machinefile _HOSTS_ _EXEC_<br>current:CORES_PER_NODE:1<br>current:MKL_TARGET_ARCH:intel64<br>current:RP_LIBS:<br><br>linux-vdso.so.1 (0x00007ffd78bac000)<br> liblapack.so.3 => /usr/lib/x86_64-linux-gnu/liblapack.so.3 (0x000015344ad82000)<br> libblas.so.3 => /usr/lib/x86_64-linux-gnu/libblas.so.3 (0x000015344ab15000)<br> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x000015344a8f6000)<br> libgfortran.so.4 => /usr/lib/x86_64-linux-gnu/libgfortran.so.4 (0x000015344a517000)<br> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000015344a179000)<br> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x0000153449d88000)<br> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x0000153449b70000)<br> /lib64/ld-linux-x86-64.so.2 (0x000015344ba2e000)<br> libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x0000153449930000)<br></div><br><div><div dir="ltr">On Thu, May 23, 2019 at 8:52 PM Indranil mal <<a href="mailto:indranil.mal@gmail.com">indranil.mal@gmail.com</a>> wrote:<br></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Thanks a lot. <br></div><div>Sir my calculations are running when I do the x lapw1 may be due to that this time is too long. <br></div><div>I have installed ifort and intel mpi mkl but could not configured that is why I am using GFORTRAN and gcc the basic gnu compiler and open blas. If you dont mind you can access my pc through team viewer. <br></div><div><br></div><div><br></div></div><br><div><div dir="ltr">On Thu, May 23, 2019 at 7:50 PM Pavel Ondračka <<a href="mailto:pavel.ondracka@email.cz">pavel.ondracka@email.cz</a>> wrote:<br></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Well,<br>
<br>
first we need to figure out why is your serial lapw so slow...<br>
You definitely don't have the libmvec patches, however almost two min<br>
runtime suggest that even your BLAS might be bad?<br>
<br>
In the test_case folder run:<br>
$ grep "TIME HAMILT" test_case.output1<br>
and post the output. Also please go to the Wien2k folder and send the<br>
output of <br>
$ cat WIEN2k_OPTION<br>
and<br>
$ ldd lapw1<br>
<br>
Next Wien2k version will have this simplified, however for now some<br>
patching needs to be to be done. The other option would be to get MKL<br>
and ifort from Intel and use it instead...<br>
<br>
Anyway if you don't want MKL, you need to download the attached patch<br>
to the SRC_lapw1 folder in Wien2k base folder.<br>
Go to the folder, and apply the patch with (you might need the patch<br>
package for that)<br>
$ patch -p1 < lapw1.patch<br>
then set the FOPT compile flags via siteconfig to: <br>
-ffree-form -O2 -ffree-line-length-none -march=native -ftree-vectorize<br>
-DHAVE_LIBMVEC -fopenmp<br>
and recompile lapw1.<br>
Now when you do again<br>
$ ldd lapw1<br>
it should show line with "libmvec.so.1 => /lib64/libmvec.so.1"<br>
<br>
Compare timings again with the test_case.<br>
Also try:<br>
$ OMP_NUM_THREADS=2 x lapw1<br>
$ OMP_NUM_THREADS=4 x lapw1<br>
<br>
And after each run show total timings as well as<br>
$ grep "TIME HAMILT" test_case.output1<br>
Hopefully, you are already linking the multithreaded Openblas (but<br>
dunno what is the Ubuntu default)...<br>
<br>
I'll help you with the parallel execution in the next step.<br>
<br>
Best regards<br>
Pavel<br>
<br>
On Thu, 2019-05-23 at 18:58 +0530, Indranil mal wrote:<br>
> Dear sir <br>
> <br>
> After running x lapw1 I got the following <br>
> <br>
> ~/test_case$ x lapw1<br>
> STOP LAPW1 END<br>
> 114.577u 0.247s 1:54.82 99.9% 0+0k 0+51864io 0pf+0w<br>
> <br>
> I am using parallel k point execution only 8 GB memory is in use and<br>
> for 100 atom (100 kpoints) calculation it is taking around 12 hours<br>
> to complete one cycle. <br>
> please help me. <br>
> <br>
> Thanking you<br>
> <br>
> Indranil <br>
> <br>
> On Thu, May 23, 2019 at 11:22 AM Pavel Ondračka <<br>
> <a href="mailto:pavel.ondracka@email.cz">pavel.ondracka@email.cz</a>> wrote:<br>
> > Hi Indranil,<br>
> > <br>
> > While the k-point parallelization is usually the most efficient <br>
> > (provided you have sufficient number of k-points) and does not need<br>
> > any<br>
> > extra libraries, for 100atoms case it might be problematic to fit<br>
> > 12<br>
> > processes into 32GB of memory. I assume you are already using it<br>
> > since<br>
> > you claim to run on two cores?<br>
> > <br>
> > Instead check what is the maximum memory requirement of lapw1 when<br>
> > run<br>
> > in serial and based on that find how much processes you can run in<br>
> > parallel, than for each place one line "1:localhost" into .machines<br>
> > file (there is no need to copy .machines from templates, or use<br>
> > random<br>
> > scripts, instead read the userguide to understand what you are<br>
> > doing,<br>
> > it will save you time in the long run). If you can run at least few<br>
> > k-<br>
> > points in parallel it might be enough to speed it up significantly.<br>
> > <br>
> > For MPI you would need openmpi-devel scalapack-devel and fftw3-<br>
> > devel<br>
> > (I'm not sure how exactly are they named on Ubuntu) packages.<br>
> > Especially the scalapack configuration could be tricky, it is<br>
> > probably<br>
> > easiest to start with lapw0 as this needs only MPI and fftw.<br>
> > <br>
> > Also based on my experience with default gfortran settings, it is<br>
> > likely that you don't have even optimized the single core<br>
> > performance,<br>
> > try to download the serial benchmark <br>
> > <a href="http://susi.theochem.tuwien.ac.at/reg_user/benchmark/test_case.tar.gz" rel="noreferrer">http://susi.theochem.tuwien.ac.at/reg_user/benchmark/test_case.tar.gz</a><br>
> > untar, run x lapw1 and report timings (on average i7 CPU it should<br>
> > take<br>
> > below 30 seconds, if it takes significantly more, you will need<br>
> > some<br>
> > more tweaks).<br>
> > <br>
> > Best regards<br>
> > Pavel<br>
> > <br>
> > On Thu, 2019-05-23 at 10:42 +0530, Dr. K. C. Bhamu wrote:<br>
> > > Hii,<br>
> > > <br>
> > > If you are doing k-point parallel calculation (having number of<br>
> > k-<br>
> > > points in IBZ more then 12) then use below script on terminal<br>
> > where<br>
> > > you want to run the calculation or use in your job script with<br>
> > -p<br>
> > > option in run(sp)_lapw (-so).<br>
> > > <br>
> > > if anyone knows how to repeat a nth line m times in a file then<br>
> > this<br>
> > > script can be changed.<br>
> > > <br>
> > > Below script simply coping machine file from temple directory and<br>
> > > updating it as per your need.<br>
> > > So you do not need copy it, open it in your favorite editor and<br>
> > do it<br>
> > > manually.<br>
> > > <br>
> > > cp $WIENROOT/SRC_templates/.machines . ; grep localhost .machines<br>
> > |<br>
> > > perl -ne 'print $_ x 6' > LOCALHOST.dat ; tail -n 2 .machines ><br>
> > > grang.dat ; sed '22,25d' .machines > MACHINE.dat ; cat<br>
> > MACHINE.dat<br>
> > > localhost.dat grang.dat > .machines ; rm LOCALHOST.dat<br>
> > MACHINE.dat<br>
> > > grang.dat<br>
> > > <br>
> > > regards<br>
> > > Bhamu<br>
> > > <br>
> > > <br>
> > > On Wed, May 22, 2019 at 10:52 PM Indranil mal <<br>
> > <a href="mailto:indranil.mal@gmail.com">indranil.mal@gmail.com</a><br>
> > > > wrote:<br>
> > > > respected sir/ Users,<br>
> > > > I am using a PC with intel i7 8th gen (with<br>
> > 12<br>
> > > > cores) 32GB RAM and 2TB HDD with UBUNTU 18.04 LTS. I have<br>
> > installed<br>
> > > > OpenBLAS-0.2.20 and using GNU FORTRAN and c compiler. I am<br>
> > trying<br>
> > > > to run a system with 100 atoms only two cores are using the<br>
> > rest of<br>
> > > > them are idle and the calculation taking a too long time. I<br>
> > have<br>
> > > > not installed mpi ScaLAPACK or elpa. Please help me what should<br>
> > I<br>
> > > > do to utilize all of the cores of my cpu.<br>
> > > > <br>
> > > > <br>
> > > > <br>
> > > > Thanking you <br>
> > > > <br>
> > > > Indranil<br>
> > > > _______________________________________________<br>
> > > > Wien mailing list<br>
> > > > <a href="mailto:Wien@zeus.theochem.tuwien.ac.at">Wien@zeus.theochem.tuwien.ac.at</a><br>
> > > > <a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien" rel="noreferrer">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a><br>
> > > > SEARCH the MAILING-LIST at: <br>
> > > > <br>
> > <a href="http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html" rel="noreferrer">http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html</a><br>
> > > <br>
> > > _______________________________________________<br>
> > > Wien mailing list<br>
> > > <a href="mailto:Wien@zeus.theochem.tuwien.ac.at">Wien@zeus.theochem.tuwien.ac.at</a><br>
> > > <a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien" rel="noreferrer">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a><br>
> > > SEARCH the MAILING-LIST at: <br>
> > > <br>
> > <a href="http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html" rel="noreferrer">http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html</a><br>
> > <br>
> > _______________________________________________<br>
> > Wien mailing list<br>
> > <a href="mailto:Wien@zeus.theochem.tuwien.ac.at">Wien@zeus.theochem.tuwien.ac.at</a><br>
> > <a href="http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien" rel="noreferrer">http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien</a><br>
> > SEARCH the MAILING-LIST at: <br>
> > <a href="http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html" rel="noreferrer">http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html</a><br>
</blockquote></div>
</blockquote></div>
</blockquote></body></html>