[Wien] Parallel calculation

[Gavin Abo]

If you are using gfortran and gcc and it helps, some of the keystrokes I 
captured are shown below from when I installed WIEN2k 21.1.  These are 
just steps I follow as a guide for getting started and for a 
configuration that gets WIEN2k on my system up and running quickly.  
After that, I usually have to go back into ./siteconfig and further 
adjust the settings to make calculations complete faster by finding 
better settings to use for my system in the documentation for the 
compilers.  For example, the gfortran documentation is at [1].  Other 
compilers, like the Intel Fortran compiler [2], should have their own 
documentation.

The mpi parallel settings configured below don't always work depending 
on the configuration that I need on my computer systems.  When that 
happens, the WIEN2k usersguide [3] and FAQ questions page [4] help with 
choosing the correct siteconfig settings.  Or past posts on parallel 
calculations in the mailing list archive contained a solution to the 
issues I encountered.

I haven't tried the SRC_mpiutil on the unsupported page [5] with WIEN2k 
21.1 but I think that was helpful when I looked at that when using a 
past WIEN2k version.

For the below, I used Open MPI [6], but you could use another mpi 
implementation [7].

[1] https://gcc.gnu.org/wiki/GFortran
[2] 
https://software.intel.com/content/www/us/en/develop/documentation/fortran-compiler-oneapi-dev-guide-and-reference/top.html
[3] http://www.wien2k.at/reg_user/textbooks/usersguide.pdf
[4] http://www.wien2k.at/reg_user/faq/pbs.html
[5] http://www.wien2k.at/reg_user/unsupported/
[6] https://www.open-mpi.org/
[7] https://en.wikipedia.org/wiki/MPICH

*Installed Ubuntu LTS*

https://help.ubuntu.com/community/Installation
https://ubuntu.com/#download

username at computername:~$ lsb_release -a
No LSB modules are available.
Distributor ID:    Ubuntu
Description:    Ubuntu 20.04.2 LTS
Release:    20.04
Codename:    focal

*Installed XCrySDen*

username at computername:~$ cd ~
username at computername:~$ sudo apt update
username at computername:~$ sudo apt install tcsh ghostscript octave 
gnuplot gnuplot-x11 make autoconf libtool perl libquadmath0 gfortran 
build-essential libglu1-mesa-dev libtogl-dev tcl-dev tk-dev libfftw3-dev 
libxmu-dev
username at computername:~$ wget 
http://www.xcrysden.org/download/xcrysden-1.6.2.tar.gz
username at computername:~$ tar xvf xcrysden-1.6.2.tar.gz
username at computername:~$ cd xcrysden-1.6.2
username at computername:~/xcrysden-1.6.2$ cp ./system/Make.sys-shared Make.sys
username at computername:~/xcrysden-1.6.2$ make all
username at computername:~/xcrysden-1.6.2$ echo 'export 
XCRYSDEN_TOPDIR=/home/username/xcrysden-1.6.2'>>~/.bashrc
username at computername:~/xcrysden-1.6.2$ echo 'export 
PATH=$PATH:$XCRYSDEN_TOPDIR'>>~/.bashrc
username at computername:~/xcrysden-1.6.2$ source ~/.bashrc

*Installed libxc*

username at computername:~/xcrysden-1.6.2$ cd ~
username at computername:~$ wget 
http://www.tddft.org/programs/libxc/down.php?file=5.1.4/libxc-5.1.4.tar.gz
username at computername:~$ tar xvf down.php\?file\=5.1.4%2Flibxc-5.1.4.tar.gz
username at computername:~$ cd libxc-5.1.4/
username at computername:~/libxc-5.1.4$ autoreconf -i --force
username at computername:~/libxc-5.1.4$ ./configure FC=gfortran CC=gcc 
--prefix=$HOME/libxc-5.1.4
username at computername:~/libxc-5.1.4$ make
username at computername:~/libxc-5.1.4$ make check
username at computername:~/libxc-5.1.4$ make install

*Installed OpenBLAS*

username at computername:~/libxc-5.1.4$ cd ~
username at computername:~$ wget 
https://github.com/xianyi/OpenBLAS/releases/download/v0.3.15/OpenBLAS-0.3.15.tar.gz
username at computername:~$ tar zxvf OpenBLAS-0.3.15.tar.gz
username at computername:~$ cd OpenBLAS-0.3.15/
username at computername:~/OpenBLAS-0.3.15$ make FC=gfortran CC=gcc
username at computername:~/OpenBLAS-0.3.15$ echo 'export 
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/username/OpenBLAS-0.3.15'>>~/.bashrc
username at computername:~/OpenBLAS-0.3.15$ source ~/.bashrc

*Installed Open MPI*

username at computername:~/OpenBLAS-0.3.15$ cd ~
username at computername:~$ wget 
https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.1.tar.gz
username at computername:~$ tar xvf openmpi-4.1.1.tar.gz
username at computername:~$ cd openmpi-4.1.1/
username at computername:~/openmpi-4.1.1$ ./configure 
--prefix=$HOME/openmpi-4.1.1
username at computername:~/openmpi-4.1.1$ make all install
username at computername:~/openmpi-4.1.1$ echo 'export 
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/username/openmpi-4.1.1/lib'>>~/.bashrc
username at computername:~/openmpi-4.1.1$ echo 'export 
PATH=$PATH:/home/username/openmpi-4.1.1/bin'>>~/.bashrc
username at computername:~/openmpi-4.1.1$ source ~/.bashrc

*Installed fftw*

username at computername:~/openmpi-4.1.1$ cd ~
username at computername:~$ wget http://www.fftw.org/fftw-3.3.9.tar.gz
username at computername:~$ tar xvf fftw-3.3.9.tar.gz
username at computername:~$ cd fftw-3.3.9/
username at computername:~/fftw-3.3.9$ ./configure FCC=gfortran CC=gcc 
MPICC=mpicc --enable-mpi --prefix=$HOME/fftw-3.3.9
username at computername:~/fftw-3.3.9$ make
username at computername:~/fftw-3.3.9$ make install

*Installed ScaLAPACK*

username at computername:~/fftw-3.3.9$ cd ~
username at computername:~$ wget 
http://www.netlib.org/scalapack/scalapack-2.1.0.tgz
username at computername:~$ tar xvf scalapack-2.1.0.tgz
username at computername:~$ cd scalapack-2.1.0/
username at computername:~/scalapack-2.1.0$ cp SLmake.inc.example SLmake.inc
username at computername:~/scalapack-2.1.0$ gedit SLmake.inc
username at computername:~/scalapack-2.1.0$ cat SLmake.inc
############################################################################
#
#  Program:         ScaLAPACK
#
#  Module:          SLmake.inc
#
#  Purpose:         Top-level Definitions
#
#  Creation date:   February 15, 2000
#
#  Modified:        October 13, 2011
#
#  Send bug reports, comments or suggestions to scalapack at cs.utk.edu
#
############################################################################
#
#  C preprocessor definitions:  set CDEFS to one of the following:
#
#     -DNoChange (fortran subprogram names are lower case without any 
suffix)
#     -DUpCase   (fortran subprogram names are upper case without any 
suffix)
#     -DAdd_     (fortran subprogram names are lower case with "_" appended)

CDEFS         = -DAdd_

#
#  The fortran and C compilers, loaders, and their flags
#

FC            = mpifort
CC            = mpicc
NOOPT         = -O0
FCFLAGS       = -O3
CCFLAGS       = -O3
FCLOADER      = $(FC)
CCLOADER      = $(CC)
FCLOADFLAGS   = $(FCFLAGS)
CCLOADFLAGS   = $(CCFLAGS)

#
#  The archiver and the flag(s) to use when building archive (library)
#  Also the ranlib routine.  If your system has no ranlib, set RANLIB = echo
#

ARCH          = ar
ARCHFLAGS     = cr
RANLIB        = ranlib

#
#  The name of the ScaLAPACK library to be created
#

SCALAPACKLIB  = libscalapack.a

#
#  BLAS, LAPACK (and possibly other) libraries needed for linking test 
programs
#

BLASLIB       = -L/home/username/OpenBLAS-0.3.15 -lopenblas
LAPACKLIB     =
LIBS          = $(LAPACKLIB) $(BLASLIB)


username at computername:~/scalapack-2.1.0$ make
username at computername:~/scalapack-2.1.0$ echo 'export 
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/username/scalapack-2.1.0'>>~/.bashrc
username at computername:~/scalapack-2.1.0$ source ~/.bashrc

*Installed WIEN2k*

Went to:

http://www.wien2k.at/reg_user/index.html

Clicked "Code download (after registration)"

Downloaded "WIEN2k_21.1.tar"

username at computername:~/scalapack-2.1.0$ cd ~
username at computername:~$ mkdir WIEN2k
username at computername:~$ cd WIEN2k/
username at computername:~/WIEN2k$ ls
WIEN2k_21.1.tar
username at computername:~/WIEN2k$ tar xvf WIEN2k_21.1.tar
username at computername:~/WIEN2k$ gunzip *.gz
username at computername:~/WIEN2k$ chmod +x ./expand_lapw
username at computername:~/WIEN2k$ export LC_NUMERIC=en_US.UTF-8
username at computername:~/WIEN2k$ ./expand_lapw
continue (y/n)
y
username at computername:~/WIEN2k$ ./siteconfig
continue or stop (c/s) c
   Selection: LG
      Your compiler: gfortran
      Your compiler: gcc
  Would you like to use LIBXC (usually not needed, ONLY for experts who 
want to play with different DFT options. It must have been installed 
before)? (y,N):
y
  Do you want to automatically search for LIBXC installations? (Y,n):
Y
  Please specify a comma separated list of directories to search! (If no 
list is entered, /usr/lib64, /usr/local and /opt will be searched as 
default):
/home/username/libxc-5.1.4
  Pick one of the following (enter the number of the line of your choice 
or 0 to manually specify a path)!:
/home/username/libxc-5.1.4/src/.libs/libxc.a
/home/username/libxc-5.1.4/lib/libxc.a
2
   Do you want to automatically search for FFTW installations? (Y,n):
Y
  Please specify a comma separated list of directories to search! (If no 
list is entered,
  /usr/lib64, /usr/local and /opt will be searched as default):
/home/username/fftw-3.3.9
  Finding the required fftw3 library-files in /home/username/fftw-3.3.9 ....

/home/username/fftw-3.3.9/.libs/libfftw3.a
/home/username/fftw-3.3.9/lib/libfftw3.a

Do you want to use a FFTW version from the list above? (Y,n):
Y
  Please enter the line number of the chosen version or enter 0 to 
manually specify your choice!:
2
  Please specify the target achitecture of your FFTW library (e.g. 
lib64) or accept present choice (enter): lib
   Please specify the name of your FFTW library or accept present choice 
(enter): fftw3
   Is this correct? (Y,n): Y
Selection: R
      Real libraries=-L/home/username/OpenBLAS-0.3.15 -lopenblas -lpthread
Selection: X
Selection:F
Enter the name of your LIBXC fortran interface!: xcf03
Selection:B

Current settings:
   M   OpenMP switch:           -fopenmp
   O   Compiler options:        -ffree-form -O2 -ftree-vectorize 
-march=native -ffree-line-length-none -ffpe-summary=none
   L   Linker Flags:            $(FOPT) -L../SRC_lib
   P   Preprocessor flags       '-DParallel'
   R   R_LIBS (LAPACK+BLAS):    -L/home/username/OpenBLAS-0.3.15 
-lopenblas -lpthread
   F   FFTW options:            -DFFTW3 -I/home/username/fftw-3.3.9/include
       FFTW-LIBS:               -L/home/username/fftw-3.3.9/lib -lfftw3
   X   LIBX options:            -DLIBXC -I/home/username/libxc-5.1.4/include
       LIBXC-LIBS:              -L/home/username/libxc-5.1.4/lib -lxcf03 
-lxc

Selection: S
   Shared Memory Architecture? (y/N):y
   Do you know/need a command to bind your jobs to specific nodes?
   (like taskset -c). Enter N / your_specific_command: N
    Do you have MPI, ScaLAPACK, ELPA, or MPI-parallel FFTW installed and 
intend
    to run finegrained parallel?
    (y/N) y
    Your compiler: mpifort
  Do you want to use a present ScaLAPACK installation? (Y,n): Y
  Please specify the target architecture of your ScaLAPACK libraries 
(e.g. intel64)!:
  Please specify the root path of your ScaLAPACK installation!:
  ScaLAPACK root: /home/username/scalapack-2.1.0
  Please specify the name of your ScaLAPACK library!:
  ScaLAPACK library: scalapack
  Please specify the root path of your BLACS installation!:
  BLACS root:
  Please specify the name of your BLACS library!:
  BLACS library:
Is this correct? (Y,n): Y
  Do you want to use ELPA? (y,N):
N
  Please enter your choice of additional libraries!:
$(R_LIBS) -lmpi
  Is this correct?  (Y,n): Y
    Please specify your parallel compiler options or accept the 
recommendations (Enter - default)!:
-ffree-form -O2 -ftree-vectorize -march=native -ffree-line-length-none 
-ffpe-summary=none

    Please specify your MPIRUN command or accept the recommendations 
(Enter - default)!:

    Please specify your parallel OMP_SWITCH (type "del" to blank it)
    or accept the recommendations (Enter - default)!:

    Please specify your MPIRUN command or accept the recommendations 
(Enter - default)!:

    Current settings:

          Parallel compiler      : mpifort
          SCALAPACK_LIBS         : -L/home/username/scalapack-2.1.0/ 
-lscalapack
          FFTW_PLIBS             : -lfftw3_mpi
          ELPA_OPT               :
          ELPA_LIBS              :
          FPOPT(par.comp.options): -ffree-form -O2 -ftree-vectorize 
-march=native -ffree-line-length-none -ffpe-summary=none
          OMP_SWITCH             : -fopenmp
          MPIRUN command         : mpirun -np _NP_ -machinefile _HOSTS_ 
_EXEC_

    parallel execution:

          RP_LIBS                : $(R_LIBS) -lmpi

Selection: S
Selection: Q
Selection: A

  Please enter the full path of the perl program: /usr/bin/perl

  Please enter the full path to your temporary directory: /tmp

username at computername:~/WIEN2k$ ./userconfig_lapw

      Specify your prefered editor (default is emacs):
      editor shall be: gedit


      Set editor to  gedit (Y/n) Y

      Specify your prefered DATA directory, where your cases should be
      stored (for  /home/username/WIEN2k, just enter RETURN key):
      DATA directory: /home/username/wiendata

      Set DATA directory to  /home/username/wiendata (Y/n) Y

      Specify your prefered scratch directory, where big case.vector files
      can be stored (Recommended is a local directory (maybe /scratch), 
not a
      NFS directory. For your working directory, just enter RETURN key):
      scratch directory:

      Set scratch directory to  working directory (Y/n) Y

      Specify your program to read pdf files (default is okular)
      (on some Linux systems use xpdf, evince, pdfstudio, ...):evince

      Set PDFREADER to  evince (Y/n) Y

WIEN2k can use OpenMP parallelization on multicore computers.
For details please read the "Parallelization section" of the Usersguide.
Your present computer has 4 cores, but more than 4 (8) cores is useless.
How many cores do you want to use by default (4):1

      Set OMP_NUM_THREADS to  1 (Y/n) Y


!!!  The following lines will be added to your .bashrc file if you 
continue !!!
      A copy of your current .bashrc will be saved under .bashrc.savelapw !

# added by WIEN2k: BEGIN
# --------------------------------------------------------
alias lsi="ls -aslp *.in*"
alias lso="ls -aslp *.output*"
alias lsd="ls -aslp *.def"
alias lsc="ls -aslp *.clm*"
alias lss="ls -aslp *.scf* */*scf"
alias lse="ls -aslp *.error"
alias LS="ls -alsp |grep /"
alias pslapw="ps -ef |grep "lapw""
alias cdw="cd /home/username/wiendata"
if [ "$OMP_NUM_THREADS" = "" ]; then export OMP_NUM_THREADS=1; fi
#export LD_LIBRARY_PATH=.....
export EDITOR="gedit"
export SCRATCH=./
if [ "$WIENROOT" = "" ]; then export WIENROOT=/home/username/WIEN2k; fi
export W2WEB_CASE_BASEDIR=/home/username/wiendata
export STRUCTEDIT_PATH=$WIENROOT/SRC_structeditor/bin
export PDFREADER=evince
export 
PATH=$WIENROOT:$STRUCTEDIT_PATH:$WIENROOT/SRC_IRelast/script-elastic:$PATH:.
export OCTAVE_EXEC_PATH=${PATH}::
export OCTAVE_PATH=${STRUCTEDIT_PATH}::

ulimit -s unlimited
alias octave="octave -p $OCTAVE_PATH"
# --------------------------------------------------------
      Do you want to continue (Y/n)? Y

      Edit .rhosts file now? (y/N) N

username at computername:~/WIEN2k$ w2web

Please enter the username: [admin]
Please enter the password: [password]
admin:password

Select the port to run on: [7890]

Please enter this system's hostname: [computername] localhost

Is this your master node?: [y] y

On 6/8/2021 8:15 AM, ben amara imen wrote:
> Dear
>
> Can someone tell me how I can install the parallel calculation for 
> Wien2k .
> Thanks in advance
>
>
> Best regards
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://zeus.theochem.tuwien.ac.at/pipermail/wien/attachments/20210608/c680f5bb/attachment.htm>