[Wien] Need extensive help for a job file for slurm job scheduler cluster

Tran, Fabien fabien.tran at tuwien.ac.at
Sun Nov 15 09:18:58 CET 2020


The probably simplest solution is to ask the system administrator to install bc on all nodes:
​​https://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/msg17679.html


From: Wien <wien-bounces at zeus.theochem.tuwien.ac.at> on behalf of Dr. K. C. Bhamu <kcbhamu85 at gmail.com>
Sent: Sunday, November 15, 2020 8:07 AM
To: A Mailing list for WIEN2k users
Subject: Re: [Wien] Need extensive help for a job file for slurm job scheduler cluster
  


Additional information (maybe this is the main cause of the lapw1 crash):
bc is only working on the head node. node11 or other clint nodes are not having bc installed.
If the bc is only the issue then is it possible to modify the job file such that it uses bc on the head node only.


Thank you
Bhamu 


On Sun, Nov 15, 2020 at 12:25 PM Dr. K. C. Bhamu <kcbhamu85 at gmail.com> wrote:
 


Dear Gavin and Prof. Marks
Thank you for your inputs.
qsub MyJobFIle.job creates the .machines file.


With the below given  job file, I could create the proper .machine files (equal to number of cores in the node and .machines file) but  lapw1  always crashes


case.dayfile is

Calculating pbe in /home/kcbhamu/work/test/pbe
on node11 with PID 9241
using WIEN2k_19.1 (Release 25/6/2019) in /home/kcbhamu/soft/w2k192


    start (Sun Nov 15 15:42:05 KST 2020) with lapw0 (40/99 to go)

    cycle 1 (Sun Nov 15 15:42:05 KST 2020) (40/99 to go)

>   lapw0   -p (15:42:05) starting parallel lapw0 at Sun Nov 15 15:42:05 KST 2020
-------- .machine0 : processors
running lapw0 in single mode
7.281u 0.272s 0:07.64 98.8% 0+0k 1000+1216io 0pf+0w
>   lapw1  -p     (15:42:13) starting parallel lapw1 at Sun Nov 15 15:42:13 KST 2020
->  starting parallel LAPW1 jobs at Sun Nov 15 15:42:13 KST 2020
running LAPW1 in parallel mode (using .machines)
16 number_of_parallel_jobs
0.200u 0.369s 0:00.59 94.9% 0+0k 208+456io 0pf+0w
error: command   /home/kcbhamu/soft/w2k192/lapw1para lapw1.def   failed

>   stop error



the job.eout file indicates below error:


But I am getting below error


bc: Command not found.
 LAPW0 END
bc: Command not found.
number_per_job: Subscript out of range.
grep: *scf1*: No such file or directory
grep: lapw2*.error: No such file or directory



.machines file is give below



1:node11
1:node11
1:node11
1:node11
1:node11
1:node11
1:node11
1:node11
1:node11
1:node11
1:node11
1:node11
1:node11
1:node11
1:node11
1:node11
granularity:1
extrafine:1





parallel_options file
setenv TASKSET "no"
if ( ! $?USE_REMOTE ) setenv USE_REMOTE 0
if ( ! $?MPI_REMOTE ) setenv MPI_REMOTE 0
setenv WIEN_GRANULARITY 1
setenv DELAY 0.1
setenv SLEEPY 1
setenv WIEN_MPIRUN "mpirun -np _NP_ -machinefile _HOSTS_ _EXEC_"
setenv CORES_PER_NODE 16



job file


#!/bin/sh
#SBATCH -J test
#SBATCH -p 52core    # THis is the name of the partition.
#SBATCH -N 1
#SBATCH -n 16
#SBATCH -o %x.o%j
#SBATCH -e %x.e%j
#export I_MPI_PMI_LIBRARY=/usr/lib64/libpmi.so 

export OMP_NUM_THREADS=16     # I have check with 1,2 4, 8 also.

# Use , as list separator
IFS=','
# Convert string to array
hcpus=($SLURM_JOB_CPUS_PER_NODE)
unset IFS

declare -a conv

# Expand compressed slurm array
for cpu in ${hcpus[@]}; do
     if [[ $cpu =~ (.*)\((.*)x\) ]]; then
# found compressed value
value=${BASH_REMATCH[1]}
factor=${BASH_REMATCH[2]}
for j in $(seq 1 $factor); do
   conv=( ${conv[*]} $value )
done
     else
conv=( ${conv[*]} $cpu )
     fi
done

# Build .machines file
rm -f .machines

nhost=0

echo ${conv[@]};

IFS=','
for node in $SLURM_NODELIST
do 
    declare -i cpuspernode=${conv[$nhost]};
    for ((i=0; i<${cpuspernode}; i++)) 
    do
echo 1:$node >> .machines
    done
    let nhost+=1
done 

echo 'granularity:1' >>.machines
echo 'extrafine:1' >>.machines


run_lapw -p





Thank you very much


Regards
Bhamu



 


On Fri, Nov 13, 2020 at 7:04 PM Gavin Abo <gsabo at crimson.ua.edu> wrote:
 
If you have a look at [1], it can be seen that different cluster systems have different commands for job submission.
I did not see it clearly shown in your post how the job was submitted, for example did you maybe use something similar to that at [2]:
$ sbatch MyJobScript.sh

What command creates your .machines file?

In your MyJobScript.sh below, I'm not seeing any lines that create a .machines file.

 MyJobScript.sh
--------------------------------------------------------------------------------------------------------
#!/bin/sh
#SBATCH -J test #job name
#SBATCH -p 44core #partition name
#SBATCH -N 1 #node
#SBATCH -n 18 #core
#SBATCH -o %x.o%j
#SBATCH -e %x.e%j
export I_MPI_PMI_LIBRARY=/usr/lib64/libpmi.so #Do not change here!!
srun ~/soft/qe66/bin/pw.x  < case.in > case.out
-------------------------------------------------------------------------------------------------------- The available jobs files on FAQs are not working. They give me
.machine0          .machines          .machines_current   files only wherein .machines has # and the other two are empty.

In the Slurm documentation at [3], it looks like there is variable for helping creating a list of nodes on the fly that would need to be written to the .machines file:
SLURM_JOB_NODELIST (and SLURM_NODELIST for backwards compatibility)

I'm not seeing this in your MyJobScript.sh like that seen in other job scripts found on the Internet, for example [4-7].

 [1] https://slurm.schedmd.com/rosetta.pdf
[2]  https://hpc-uit.readthedocs.io/en/latest/jobs/examples.html
[3] https://slurm.schedmd.com/sbatch.html
[4]  https://itp.uni-frankfurt.de/wiki-it/index.php/Wien2k
[5]  https://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/msg15511.html
[6]  https://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/msg07097.html
[7]  https://www.nsc.liu.se/software/installed/tetralith/wien2k/


On 11/13/2020 3:37 AM, Laurence Marks wrote:
 

N.B., example mid-term questions:
1. What SBATCH command will give you 3 nodes?
2. What command creates your .machines file?
3. What are your fastest and slowest nodes?
4. Which nodes have the best communications.


N.B., please don't post your answers -- just understand!


_____
Professor Laurence Marks
"Research is to see what everybody else has seen, and to think what nobody else has thought", Albert Szent-Gyorgi
www.numis.northwestern.edu 


On Fri, Nov 13, 2020, 04:21 Laurence Marks <laurence.marks at gmail.com> wrote:
 
Much of what you are requesting is problem/cluster specific, so there is no magic answer -- it will vary. Suggestions:
1) Read the UG sections on .machines and parallel operation.
2) Read the man page for your cluster job command (srun)
3) Reread the UG sections.
4) Read the example scripts, and understand (lookup) all the commands so you know what they are doing.


It is really not that complicated. If you cannot master this by yourself, I will wonder whether you are in the right profession.


_____
Professor Laurence Marks
"Research is to see what everybody else has seen, and to think what nobody else has thought", Albert Szent-Gyorgi
www.numis.northwestern.edu  


On Fri, Nov 13, 2020, 03:24 Dr. K. C. Bhamu <kcbhamu85 at gmail.com> wrote:
 

Dear All


I need your extensive help.
I have tried to provide full details that can help you understand my requirement. In case I have missed something, please let me know.


I am looking for a job file for our cluster. The available jobs files on FAQs are not working. They give me
.machine0          .machines          .machines_current   files only wherein .machines has # and the other two are empty.



The script that is working fine for Quantum Espresso for 44core partition is below
#!/bin/sh
#SBATCH -J test #job name
#SBATCH -p 44core #partition name
#SBATCH -N 1 #node
#SBATCH -n 18 #core
#SBATCH -o %x.o%j
#SBATCH -e %x.e%j
export I_MPI_PMI_LIBRARY=/usr/lib64/libpmi.so #Do not change here!!
srun ~/soft/qe66/bin/pw.x  <  case.in > case.out



I have compiled Wien2k_19.2 on the Centos queuing system which has the head node of Centos kernel Linux 3.10.0-1127.19.1.el7.x86_64.


I used compilers_and_libraries_2020.2.254 , fftw-3.3.8 , libxc-4.34 for the installation.


The details of the nodes that I can use are as follows (I can login into these nodes with my user password):
NODELIST   NODES PARTITION       STATE CPUS    S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON               
elpidos        1    master        idle 4       4:1:1  15787        0      1   (null) none                
node01         1    72core   allocated 72     72:1:1 515683        0      1   (null) none                
node02         1    72core   allocated 72     72:1:1 257651        0      1   (null) none                
node03         1    72core   allocated 72     72:1:1 257651        0      1   (null) none                
node09         1    44core       mixed 44     44:1:1 128650        0      1   (null) none                
node10         1    44core       mixed 44     44:1:1 128649        0      1   (null) none                
node11         1   52core*   allocated 52     52:1:1 191932        0      1   (null) none                
node12         1   52core*   allocated 52     52:1:1 191932        0      1   (null) none                 



The other nodes have a mixture of the kernel as below.


   OS=Linux 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020
   OS=Linux 3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25 17:23:54 UTC 2020 
   OS=Linux 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 
   OS=Linux 3.10.0-957.12.2.el7.x86_64 #1 SMP Tue May 14 21:24:32 UTC 2019 



Your extensive help will improve my research productivity.


Thank you very much.
Regards
Bhamu           _______________________________________________
Wien mailing list
Wien at zeus.theochem.tuwien.ac.at
http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
SEARCH the MAILING-LIST at:   http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html
        


More information about the Wien mailing list