[Wien] Need extensive help for a job file for slurm job scheduler cluster
Tran, Fabien
fabien.tran at tuwien.ac.at
Sun Nov 15 09:18:58 CET 2020
The probably simplest solution is to ask the system administrator to install bc on all nodes:
https://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/msg17679.html
From: Wien <wien-bounces at zeus.theochem.tuwien.ac.at> on behalf of Dr. K. C. Bhamu <kcbhamu85 at gmail.com>
Sent: Sunday, November 15, 2020 8:07 AM
To: A Mailing list for WIEN2k users
Subject: Re: [Wien] Need extensive help for a job file for slurm job scheduler cluster
Additional information (maybe this is the main cause of the lapw1 crash):
bc is only working on the head node. node11 or other clint nodes are not having bc installed.
If the bc is only the issue then is it possible to modify the job file such that it uses bc on the head node only.
Thank you
Bhamu
On Sun, Nov 15, 2020 at 12:25 PM Dr. K. C. Bhamu <kcbhamu85 at gmail.com> wrote:
Dear Gavin and Prof. Marks
Thank you for your inputs.
qsub MyJobFIle.job creates the .machines file.
With the below given job file, I could create the proper .machine files (equal to number of cores in the node and .machines file) but lapw1 always crashes
case.dayfile is
Calculating pbe in /home/kcbhamu/work/test/pbe
on node11 with PID 9241
using WIEN2k_19.1 (Release 25/6/2019) in /home/kcbhamu/soft/w2k192
start (Sun Nov 15 15:42:05 KST 2020) with lapw0 (40/99 to go)
cycle 1 (Sun Nov 15 15:42:05 KST 2020) (40/99 to go)
> lapw0 -p (15:42:05) starting parallel lapw0 at Sun Nov 15 15:42:05 KST 2020
-------- .machine0 : processors
running lapw0 in single mode
7.281u 0.272s 0:07.64 98.8% 0+0k 1000+1216io 0pf+0w
> lapw1 -p (15:42:13) starting parallel lapw1 at Sun Nov 15 15:42:13 KST 2020
-> starting parallel LAPW1 jobs at Sun Nov 15 15:42:13 KST 2020
running LAPW1 in parallel mode (using .machines)
16 number_of_parallel_jobs
0.200u 0.369s 0:00.59 94.9% 0+0k 208+456io 0pf+0w
error: command /home/kcbhamu/soft/w2k192/lapw1para lapw1.def failed
> stop error
the job.eout file indicates below error:
But I am getting below error
bc: Command not found.
LAPW0 END
bc: Command not found.
number_per_job: Subscript out of range.
grep: *scf1*: No such file or directory
grep: lapw2*.error: No such file or directory
.machines file is give below
1:node11
1:node11
1:node11
1:node11
1:node11
1:node11
1:node11
1:node11
1:node11
1:node11
1:node11
1:node11
1:node11
1:node11
1:node11
1:node11
granularity:1
extrafine:1
parallel_options file
setenv TASKSET "no"
if ( ! $?USE_REMOTE ) setenv USE_REMOTE 0
if ( ! $?MPI_REMOTE ) setenv MPI_REMOTE 0
setenv WIEN_GRANULARITY 1
setenv DELAY 0.1
setenv SLEEPY 1
setenv WIEN_MPIRUN "mpirun -np _NP_ -machinefile _HOSTS_ _EXEC_"
setenv CORES_PER_NODE 16
job file
#!/bin/sh
#SBATCH -J test
#SBATCH -p 52core # THis is the name of the partition.
#SBATCH -N 1
#SBATCH -n 16
#SBATCH -o %x.o%j
#SBATCH -e %x.e%j
#export I_MPI_PMI_LIBRARY=/usr/lib64/libpmi.so
export OMP_NUM_THREADS=16 # I have check with 1,2 4, 8 also.
# Use , as list separator
IFS=','
# Convert string to array
hcpus=($SLURM_JOB_CPUS_PER_NODE)
unset IFS
declare -a conv
# Expand compressed slurm array
for cpu in ${hcpus[@]}; do
if [[ $cpu =~ (.*)\((.*)x\) ]]; then
# found compressed value
value=${BASH_REMATCH[1]}
factor=${BASH_REMATCH[2]}
for j in $(seq 1 $factor); do
conv=( ${conv[*]} $value )
done
else
conv=( ${conv[*]} $cpu )
fi
done
# Build .machines file
rm -f .machines
nhost=0
echo ${conv[@]};
IFS=','
for node in $SLURM_NODELIST
do
declare -i cpuspernode=${conv[$nhost]};
for ((i=0; i<${cpuspernode}; i++))
do
echo 1:$node >> .machines
done
let nhost+=1
done
echo 'granularity:1' >>.machines
echo 'extrafine:1' >>.machines
run_lapw -p
Thank you very much
Regards
Bhamu
On Fri, Nov 13, 2020 at 7:04 PM Gavin Abo <gsabo at crimson.ua.edu> wrote:
If you have a look at [1], it can be seen that different cluster systems have different commands for job submission.
I did not see it clearly shown in your post how the job was submitted, for example did you maybe use something similar to that at [2]:
$ sbatch MyJobScript.sh
What command creates your .machines file?
In your MyJobScript.sh below, I'm not seeing any lines that create a .machines file.
MyJobScript.sh
--------------------------------------------------------------------------------------------------------
#!/bin/sh
#SBATCH -J test #job name
#SBATCH -p 44core #partition name
#SBATCH -N 1 #node
#SBATCH -n 18 #core
#SBATCH -o %x.o%j
#SBATCH -e %x.e%j
export I_MPI_PMI_LIBRARY=/usr/lib64/libpmi.so #Do not change here!!
srun ~/soft/qe66/bin/pw.x < case.in > case.out
-------------------------------------------------------------------------------------------------------- The available jobs files on FAQs are not working. They give me
.machine0 .machines .machines_current files only wherein .machines has # and the other two are empty.
In the Slurm documentation at [3], it looks like there is variable for helping creating a list of nodes on the fly that would need to be written to the .machines file:
SLURM_JOB_NODELIST (and SLURM_NODELIST for backwards compatibility)
I'm not seeing this in your MyJobScript.sh like that seen in other job scripts found on the Internet, for example [4-7].
[1] https://slurm.schedmd.com/rosetta.pdf
[2] https://hpc-uit.readthedocs.io/en/latest/jobs/examples.html
[3] https://slurm.schedmd.com/sbatch.html
[4] https://itp.uni-frankfurt.de/wiki-it/index.php/Wien2k
[5] https://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/msg15511.html
[6] https://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/msg07097.html
[7] https://www.nsc.liu.se/software/installed/tetralith/wien2k/
On 11/13/2020 3:37 AM, Laurence Marks wrote:
N.B., example mid-term questions:
1. What SBATCH command will give you 3 nodes?
2. What command creates your .machines file?
3. What are your fastest and slowest nodes?
4. Which nodes have the best communications.
N.B., please don't post your answers -- just understand!
_____
Professor Laurence Marks
"Research is to see what everybody else has seen, and to think what nobody else has thought", Albert Szent-Gyorgi
www.numis.northwestern.edu
On Fri, Nov 13, 2020, 04:21 Laurence Marks <laurence.marks at gmail.com> wrote:
Much of what you are requesting is problem/cluster specific, so there is no magic answer -- it will vary. Suggestions:
1) Read the UG sections on .machines and parallel operation.
2) Read the man page for your cluster job command (srun)
3) Reread the UG sections.
4) Read the example scripts, and understand (lookup) all the commands so you know what they are doing.
It is really not that complicated. If you cannot master this by yourself, I will wonder whether you are in the right profession.
_____
Professor Laurence Marks
"Research is to see what everybody else has seen, and to think what nobody else has thought", Albert Szent-Gyorgi
www.numis.northwestern.edu
On Fri, Nov 13, 2020, 03:24 Dr. K. C. Bhamu <kcbhamu85 at gmail.com> wrote:
Dear All
I need your extensive help.
I have tried to provide full details that can help you understand my requirement. In case I have missed something, please let me know.
I am looking for a job file for our cluster. The available jobs files on FAQs are not working. They give me
.machine0 .machines .machines_current files only wherein .machines has # and the other two are empty.
The script that is working fine for Quantum Espresso for 44core partition is below
#!/bin/sh
#SBATCH -J test #job name
#SBATCH -p 44core #partition name
#SBATCH -N 1 #node
#SBATCH -n 18 #core
#SBATCH -o %x.o%j
#SBATCH -e %x.e%j
export I_MPI_PMI_LIBRARY=/usr/lib64/libpmi.so #Do not change here!!
srun ~/soft/qe66/bin/pw.x < case.in > case.out
I have compiled Wien2k_19.2 on the Centos queuing system which has the head node of Centos kernel Linux 3.10.0-1127.19.1.el7.x86_64.
I used compilers_and_libraries_2020.2.254 , fftw-3.3.8 , libxc-4.34 for the installation.
The details of the nodes that I can use are as follows (I can login into these nodes with my user password):
NODELIST NODES PARTITION STATE CPUS S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON
elpidos 1 master idle 4 4:1:1 15787 0 1 (null) none
node01 1 72core allocated 72 72:1:1 515683 0 1 (null) none
node02 1 72core allocated 72 72:1:1 257651 0 1 (null) none
node03 1 72core allocated 72 72:1:1 257651 0 1 (null) none
node09 1 44core mixed 44 44:1:1 128650 0 1 (null) none
node10 1 44core mixed 44 44:1:1 128649 0 1 (null) none
node11 1 52core* allocated 52 52:1:1 191932 0 1 (null) none
node12 1 52core* allocated 52 52:1:1 191932 0 1 (null) none
The other nodes have a mixture of the kernel as below.
OS=Linux 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020
OS=Linux 3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25 17:23:54 UTC 2020
OS=Linux 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016
OS=Linux 3.10.0-957.12.2.el7.x86_64 #1 SMP Tue May 14 21:24:32 UTC 2019
Your extensive help will improve my research productivity.
Thank you very much.
Regards
Bhamu _______________________________________________
Wien mailing list
Wien at zeus.theochem.tuwien.ac.at
http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
SEARCH the MAILING-LIST at: http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html
More information about the Wien
mailing list