1slurm
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
1slurm [2020/03/30 10:28] – admin | 1slurm [2020/06/26 10:38] (current) – admin | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | !!!! WORK IN PROGRESS !!!!! | + | |
- | ====== | + | ====== |
Our cluster runs [[https:// | Our cluster runs [[https:// | ||
- | The queuing system give you access to computers owned by LCM, LTHC, LTHI, LINX and IC Faculty; sharing the computational resources among as many groups as possible will result in a more efficient use of the resources (including the electric power). | + | The queuing system give you access to computers owned by LCM, LTHC, LTHI, LINX and IC Faculty; sharing the computational resources among as many groups as possible will result in a more efficient use of the resources (including the electric power), you can take advantage of many more machines for your urgent calculations and get results faster. |
- | As user you can take advantage of many more machines for your urgent calculations and get results faster. On the other hand, since the machines | + | On the other hand, since the machines |
- | We have configured the system almost | + | We have configured the system |
- number of CPU/cores: you must indicate the correct number of cores you're going to use; | - number of CPU/cores: you must indicate the correct number of cores you're going to use; | ||
- Megabytes/ | - Megabytes/ | ||
- Time for the execution: if your job is not completed by the indicated time, it will be automatically terminated; | - Time for the execution: if your job is not completed by the indicated time, it will be automatically terminated; | ||
- | you can find better and more complete guides on how to use S.L.U.R.M. control commands on internet; e.g: | + | here we provide just a fast and dirty guide for the most basic commands/ |
- [[https:// | - [[https:// | ||
- [[https:// | - [[https:// | ||
- [[https:// | - [[https:// | ||
- | here we provide just a fast and dirty guide for the most basic commands/ | ||
==== partitions (a.k.a. queues) ==== | ==== partitions (a.k.a. queues) ==== | ||
- | If you used other types of cluster | + | If you used other types of cluster |
===== Mini User Guide ===== | ===== Mini User Guide ===== | ||
- | The 3 most used commands are: | + | The most used/ |
- | - '' | + | - '' |
- | - '' | + | - '' |
- | - '' | + | - '' |
- | - '' | + | - '' |
- | * '' | + | * '' |
< | < | ||
$ sinfo | $ sinfo | ||
Line 49: | Line 48: | ||
here you can see that the command provides the ID of the jobs, the PARTITION used to run the jobs (hence the nodes where these jobs will run), the NAME assigned to the jobs, the name of the USER that submitted the jobs, the STATUS of the job (R=Run, PD=Waiting), | here you can see that the command provides the ID of the jobs, the PARTITION used to run the jobs (hence the nodes where these jobs will run), the NAME assigned to the jobs, the name of the USER that submitted the jobs, the STATUS of the job (R=Run, PD=Waiting), | ||
- | * '' | + | * '' |
- | Once a job is submitted (and accepted by the cluster, you'll receive the ID assigned to the job: | + | Once a job is submitted (and accepted by the cluster), you'll receive the ID assigned to the job: |
< | < | ||
$ sbatch sheepit.slurm | $ sbatch sheepit.slurm | ||
Submitted batch job 552 | Submitted batch job 552 | ||
</ | </ | ||
- | * '' | + | * '' |
* '' | * '' | ||
Line 75: | Line 74: | ||
</ | </ | ||
- | === Scripts (used with **sbatch**) === | + | === Scripts (used with sbatch) === |
- | It is convenient to write the job script in a file not only because in this way the script can be reused, but also because it is also possible to set '' | + | It is convenient to write the job script in a file not only because in this way the script can be reused, but also because it is also possible to set '' |
< | < | ||
$ cat sheepit.slurm | $ cat sheepit.slurm | ||
Line 102: | Line 101: | ||
srun sleep 60 | srun sleep 60 | ||
echo " | echo " | ||
+ | |||
+ | |||
</ | </ | ||
+ | < | ||
+ | At the beginning of the file, you can read the line ''# | ||
+ | </ | ||
Inside a script, all the line that starts with the '#' | Inside a script, all the line that starts with the '#' | ||
The example above instruct the queuing system to: | The example above instruct the queuing system to: | ||
- | * ''# | + | * ''# |
- | * ''# | + | * ''# |
- | * ''# | + | * ''# |
- | * ''# | + | * ''# |
- | * ''# | + | * ''# |
* ''# | * ''# | ||
- | * ''# | + | * ''# |
- | * ''# | + | * ''# |
- | * ''# | + | * ''# |
- | * ''# | + | * ''# |
At the moment we have defined these resources: | At the moment we have defined these resources: | ||
Line 122: | Line 126: | ||
and these properties/ | and these properties/ | ||
- | * '' | + | * '' |
- | * '' | + | * '' |
- | * '' | + | * '' |
- | * '' | + | * '' |
- | * '' | + | * '' |
- | * '' | + | * '' |
+ | * '' | ||
< | < | ||
Please pay attention that '' | Please pay attention that '' | ||
- | * ''# | + | * ''# |
- | * ''# | + | * ''# |
</ | </ | ||
<note important> | <note important> | ||
- | It is **mandatory** to specify at least the estimated run time of the job and the memory needed | + | It is **mandatory** to specify at least the estimated run time of the job and the memory needed, so the scheduler can optimize the nodes/ |
Please keep in mind that longer jobs are less likely to enter the queue when the cluster load is high. Therefore, don't be lazy and do not always ask for // | Please keep in mind that longer jobs are less likely to enter the queue when the cluster load is high. Therefore, don't be lazy and do not always ask for // | ||
Line 143: | Line 148: | ||
Here you can find some useful sbatch script that can be used as starting point | Here you can find some useful sbatch script that can be used as starting point | ||
^Script | ^Script | ||
- | |{{base2.pbs|Base example script}} contains most of the useful options|qsub [qsub options] base.pbs| | + | |{{base.slurm|Base example script}} contains most of the useful options|sbatch |
- | |{{matlab.pbs|Script example for running matlab computations}}|qsub -l nodes=1: | + | |{{matlab.slurm|Script example for running matlab computations}}|sbatch |
- | |{{mathematica.pbs|Script example for running Mathematica computations}}|qsub [qsub options] mathematica.pbs| | + | |{{mathematica.slurm|Script example for running Mathematica computations}}|sbatch |
- | |{{wine.pbs|Script example for windows programs (executed under wine)}}|qsub [qsub options] wine.pbs| | + | |{{wine.slurm|Script example for windows programs (executed under wine)}}|sbatch |
\\ | \\ | ||
- | The shell running the pbs script will have access to various variables that might be usefull: | + | |
- | * '' | + | The shell running the sbatch |
- | * '' | + | |
- | * '' | + | |
- | * '' | + | |
- | * '' | + | |
See the man page for more details. | See the man page for more details. | ||
- | |||
- | ==== Making your script cross platform ==== | ||
- | Presently, we have only 64 bit compute nodes. If you need to compile for 32 bit platforms, in principle, 64 bit nodes can run 32 bit code out of the box. In reality, there might be problems due to missing or incompatible library. | ||
- | An easy solution for taking advantage both of the full set of architecture and also of the optimized 64 code on 64 bit machines is the following (suggested by Alipour Masoud): | ||
- | |||
- | - Compile two version of your code (32 and 64 bit); | ||
- | - name the two executables (32 and 64 bit) as '' | ||
- | - in your pbs script use '' | ||
- | |||
- | |||
- | ==== qdel ==== | ||
- | |||
- | When you submit a job, you receive from the system a number that is used as reference to the job. to delete the job all you have to do is launch the qdel command followed by the job number you want to delete.\\ | ||
- | < | ||
- | |||
- | $ qdel 236 | ||
- | |||
- | </ | ||
- | |||
- | You can also indicate more than one job number: | ||
- | < | ||
- | |||
- | $ qdel 236 237 241 | ||
- | |||
- | </ | ||
- | |||
- | ==== BUG ==== | ||
- | |||
- | There is a bug in pbs that appears some time when the server would like to stop a running job but the node where the job is running does not respond (e.g. it did crash). When this happens, the server starts to send you a lot of identical mail messages telling you that it had to kill your job because it exceeded the time limit. If you start to receive the same message over and over about the same JOB ID, please contact your sys admin. Thanks. | ||
===== Tips and Tricks ===== | ===== Tips and Tricks ===== | ||
=== Delete all queued jobs === | === Delete all queued jobs === | ||
< | < | ||
- | qstat -u $(whoami) -n1 | grep " | + | |
+ | squeue | ||
</ | </ | ||
- | === A script that run as long as possible === | ||
- | Here is a short script that can be useful in those cases where you have the same | ||
- | calculation to run many times (e.g. for collecting statistics). | ||
- | |||
- | Since the machines are different and take different time to run the program, one usually | ||
- | allocates the time needed by the slowest machine even if on the fastest machine the actual | ||
- | running time would be 1/10 of the requested one. | ||
- | |||
- | The following script will keep running your program until there is time left. It will use | ||
- | the time needed to run 1 iteration to decide if another one can be ran. | ||
- | |||
- | < | ||
- | qstat=/ | ||
- | jobid=${PBS_JOBID%%.*} | ||
- | |||
- | # check how much time is left and set the " | ||
- | checktime() { | ||
- | if [ -x $qstat ] ; then | ||
- | times=$(qstat -n1 $jobid | tail -n 1) | ||
- | let tend=$(echo $times | awk ' | ||
- | let tnow=$(echo $times | awk ' | ||
- | let trem=$tend-$tnow | ||
- | let tmin=$tnow/ | ||
- | if [ $trem -ge $tmin ] ; then | ||
- | moretime=" | ||
- | else | ||
- | moretime=" | ||
- | fi | ||
- | else | ||
- | # cannot say => random guess | ||
- | moretime=" | ||
- | fi | ||
- | } | ||
- | |||
- | # Execute a task as many times as possible. | ||
- | let niter=0; | ||
- | moretime=" | ||
- | while [ " | ||
- | # run your program here | ||
- | ./ | ||
- | let niter=$niter+1 | ||
- | checktime | ||
- | done | ||
- | </ | ||
1slurm.1585564105.txt.gz · Last modified: 2020/03/30 10:28 by admin