slurm
no way to compare when less than two revisions
Differences
This shows you the differences between two versions of the page.
— | slurm [2020/03/02 17:29] (current) – created admin | ||
---|---|---|---|
Line 1: | Line 1: | ||
+ | work in progress | ||
+ | ====== S.L.U.R.M. ====== | ||
+ | ===== Workload Manager ===== | ||
+ | |||
+ | Our cluster runs [[https:// | ||
+ | It is **mandatory** to use this system for running long batch jobs: interactive calculations left running in the workstations are not tolerated. | ||
+ | |||
+ | The queuing system give you access to computers owned by LCM, LTHC, LTHI, LINX and IC Faculty for a total of approximately 500 cores. Sharing the computational resources among as many groups as possible will result in a more efficient use of the resources (included the electric power). A larger cluster not only have an improved average throughput, but it is also better suited to respond to peak requests. | ||
+ | |||
+ | As user you can take advantage of many more machines for your urgent calculations and get results faster. On the other hand, since the machines your are using are not always owned by your group, try to be as fair as possible and respect the needs of other users: if you notice that the cluster is overloaded (using the commands '' | ||
+ | |||
+ | We have configured the system almost without access restriction because the queuing system can make a more efficient use of the cluster if it does not have to satisfy too many constraints. Please don't force us to introduce limitations such as, for example, reducing the maximum number of jobs executed per user. | ||
+ | < | ||
+ | As user practice showed us, we had been forced to introduce some limitations: | ||
+ | - The maximum number of jobs per user is between 130 and 150, depending on other resource you requests. this limit can be varied depending on the load of the cluster, please ask your sysadmin for such changes. | ||
+ | - It's mandatory to specify how much memory your jobs will need: if you don't specify it your job will be executed on computer with small amount of memory. | ||
+ | - It's mandatory to specify how much time your job will need to complete: if you don't specify the time needed the execution of the job will be terminated by force after one hour. | ||
+ | - Jobs that need to run for more than 120 hours have less precedence over other jobs. | ||
+ | </ | ||
+ | |||
+ | ===== Mini User Guide ===== | ||
+ | |||
+ | The 3 most used commands are: | ||
+ | - '' | ||
+ | - '' | ||
+ | - '' | ||
+ | |||
+ | ==== qstat ==== | ||
+ | * '' | ||
+ | < | ||
+ | $ qstat -q | ||
+ | |||
+ | server: pbs | ||
+ | |||
+ | Queue Memory CPU Time Walltime Node Run Que Lm State | ||
+ | ---------------- ------ -------- -------- ---- --- --- -- ----- | ||
+ | long | ||
+ | default | ||
+ | short -- | ||
+ | batch -- -- | ||
+ | algo | ||
+ | ----- ----- | ||
+ | | ||
+ | </ | ||
+ | * '' | ||
+ | < | ||
+ | $ qstat -a | ||
+ | |||
+ | licossrv4.epfl.ch: | ||
+ | | ||
+ | Job ID | ||
+ | -------------------- -------- -------- ---------- ------ ----- --- ------ ----- - ----- | ||
+ | 146.licossrv4.epfl.c damir batch STDIN 3980 | ||
+ | 147.licossrv4.epfl.c damir batch STDIN 3998 | ||
+ | 148.licossrv4.epfl.c damir batch STDIN | ||
+ | 149.licossrv4.epfl.c damir batch STDIN | ||
+ | 150.licossrv4.epfl.c damir batch STDIN | ||
+ | 151.licossrv4.epfl.c damir batch STDIN | ||
+ | 152.licossrv4.epfl.c damir batch STDIN | ||
+ | 153.licossrv4.epfl.c damir batch STDIN | ||
+ | 154.licossrv4.epfl.c damir batch STDIN | ||
+ | 155.licossrv4.epfl.c cangiani batch STDIN | ||
+ | 156.licossrv4.epfl.c cangiani batch STDIN | ||
+ | 157.licossrv4.epfl.c cangiani batch STDIN | ||
+ | 158.licossrv4.epfl.c cangiani batch STDIN | ||
+ | 159.licossrv4.epfl.c cangiani batch STDIN | ||
+ | 160.licossrv4.epfl.c cangiani batch STDIN | ||
+ | 161.licossrv4.epfl.c cangiani batch STDIN | ||
+ | 162.licossrv4.epfl.c cangiani batch STDIN | ||
+ | 163.licossrv4.epfl.c cangiani batch STDIN | ||
+ | 164.licossrv4.epfl.c cangiani batch STDIN | ||
+ | |||
+ | $ qstat -n1 | ||
+ | |||
+ | licossrv4.epfl.ch: | ||
+ | | ||
+ | Job ID | ||
+ | -------------------- -------- -------- ---------- ------ ----- --- ------ ----- - ----- | ||
+ | 165.licossrv4.epfl.c damir batch STDIN 4522 | ||
+ | 166.licossrv4.epfl.c damir batch STDIN 4549 | ||
+ | 167.licossrv4.epfl.c damir batch STDIN | ||
+ | 168.licossrv4.epfl.c damir batch STDIN | ||
+ | 169.licossrv4.epfl.c damir batch STDIN | ||
+ | 170.licossrv4.epfl.c damir batch STDIN | ||
+ | 171.licossrv4.epfl.c damir batch STDIN | ||
+ | 172.licossrv4.epfl.c damir batch STDIN | ||
+ | 173.licossrv4.epfl.c damir batch STDIN | ||
+ | 174.licossrv4.epfl.c cangiani batch STDIN | ||
+ | 175.licossrv4.epfl.c cangiani batch STDIN | ||
+ | 176.licossrv4.epfl.c cangiani batch STDIN | ||
+ | 177.licossrv4.epfl.c cangiani batch STDIN | ||
+ | 178.licossrv4.epfl.c cangiani batch STDIN | ||
+ | 179.licossrv4.epfl.c cangiani batch STDIN | ||
+ | 180.licossrv4.epfl.c cangiani batch STDIN | ||
+ | 181.licossrv4.epfl.c cangiani batch STDIN | ||
+ | 182.licossrv4.epfl.c cangiani batch STDIN | ||
+ | 183.licossrv4.epfl.c cangiani batch STDIN | ||
+ | </ | ||
+ | |||
+ | ==== qsub ==== | ||
+ | Qsub is used to submit jobs. Jobs are nothing else than short scripts where the program to be executed is launched. The easiest job is something like the following | ||
+ | < | ||
+ | $ echo "cd myProject/ | ||
+ | 188.licossrv4.epfl.ch | ||
+ | </ | ||
+ | This will change to the directory '' | ||
+ | |||
+ | === Scripts === | ||
+ | |||
+ | It is convenient to write the job script in a file not only because in this way the script can be reused, but also because it is also possible to set '' | ||
+ | < | ||
+ | $ cat myScript.sh | ||
+ | # lines starting with #PBS are directives for qsub | ||
+ | #PBS -j oe | ||
+ | #PBS -o myScript.out | ||
+ | #PBS -l nodes=1: | ||
+ | |||
+ | cd bin | ||
+ | ./bogo | ||
+ | |||
+ | </ | ||
+ | |||
+ | Inside a script, all the line that starts with the '#' | ||
+ | The example above instruct the queuing system to: | ||
+ | * ''# | ||
+ | * ''# | ||
+ | * ''# | ||
+ | * ''# | ||
+ | \\ | ||
+ | Many options are available for the qsub command. The most important are the following: | ||
+ | * '' | ||
+ | * '' | ||
+ | * '' | ||
+ | * '' | ||
+ | * '' | ||
+ | * '' | ||
+ | The properties available on the various nodes can be listed with the '' | ||
+ | For the moment we have defined these properties: | ||
+ | * '' | ||
+ | * '' | ||
+ | * '' | ||
+ | * '' | ||
+ | * '' | ||
+ | * '' | ||
+ | Example **qsub -l nodes=1: | ||
+ | |||
+ | <note important> | ||
+ | It is **mandatory** to specify at least the estimated run time of the job and the memory needed by so that the scheduler can optimize the machines usage and the overall cluster throughput. If your job will pass the limits you fixed, it will be automatically killed by the cluster manager. | ||
+ | |||
+ | By default, if no time limit is specified, the job is sent to the '' | ||
+ | |||
+ | Please keep in mind that longer jobs are less likely to enter the queue when the cluster load is high. Therefore, don't be lazy and do not always ask for // | ||
+ | </ | ||
+ | |||
+ | * '' | ||
+ | The '' | ||
+ | If the month, '' | ||
+ | \\ | ||
+ | Here you can find some useful pbs script that can be used as starting point | ||
+ | ^Script | ||
+ | |{{base2.pbs|Base example script}} contains most of the useful options|qsub [qsub options] base.pbs| | ||
+ | |{{matlab.pbs|Script example for running matlab computations}}|qsub -l nodes=1: | ||
+ | |{{mathematica.pbs|Script example for running Mathematica computations}}|qsub [qsub options] mathematica.pbs| | ||
+ | |{{wine.pbs|Script example for windows programs (executed under wine)}}|qsub [qsub options] wine.pbs| | ||
+ | |||
+ | \\ | ||
+ | |||
+ | The shell running the pbs script will have access to various variables that might be usefull: | ||
+ | * '' | ||
+ | * '' | ||
+ | * '' | ||
+ | * '' | ||
+ | * '' | ||
+ | |||
+ | See the man page for more details. | ||
+ | |||
+ | ==== Making your script cross platform ==== | ||
+ | Presently, we have only 64 bit compute nodes. If you need to compile for 32 bit platforms, in principle, 64 bit nodes can run 32 bit code out of the box. In reality, there might be problems due to missing or incompatible library. | ||
+ | An easy solution for taking advantage both of the full set of architecture and also of the optimized 64 code on 64 bit machines is the following (suggested by Alipour Masoud): | ||
+ | |||
+ | - Compile two version of your code (32 and 64 bit); | ||
+ | - name the two executables (32 and 64 bit) as '' | ||
+ | - in your pbs script use '' | ||
+ | |||
+ | |||
+ | ==== qdel ==== | ||
+ | |||
+ | When you submit a job, you receive from the system a number that is used as reference to the job. to delete the job all you have to do is launch the qdel command followed by the job number you want to delete.\\ | ||
+ | < | ||
+ | |||
+ | $ qdel 236 | ||
+ | |||
+ | </ | ||
+ | |||
+ | You can also indicate more than one job number: | ||
+ | < | ||
+ | |||
+ | $ qdel 236 237 241 | ||
+ | |||
+ | </ | ||
+ | |||
+ | ==== BUG ==== | ||
+ | |||
+ | There is a bug in pbs that appears some time when the server would like to stop a running job but the node where the job is running does not respond (e.g. it did crash). When this happens, the server starts to send you a lot of identical mail messages telling you that it had to kill your job because it exceeded the time limit. If you start to receive the same message over and over about the same JOB ID, please contact your sys admin. Thanks. | ||
+ | |||
+ | ===== Tips and Tricks ===== | ||
+ | === Delete all queued jobs === | ||
+ | < | ||
+ | qstat -u $(whoami) -n1 | grep " | ||
+ | </ | ||
+ | |||
+ | === A script that run as long as possible === | ||
+ | Here is a short script that can be useful in those cases where you have the same | ||
+ | calculation to run many times (e.g. for collecting statistics). | ||
+ | |||
+ | Since the machines are different and take different time to run the program, one usually | ||
+ | allocates the time needed by the slowest machine even if on the fastest machine the actual | ||
+ | running time would be 1/10 of the requested one. | ||
+ | |||
+ | The following script will keep running your program until there is time left. It will use | ||
+ | the time needed to run 1 iteration to decide if another one can be ran. | ||
+ | |||
+ | < | ||
+ | qstat=/ | ||
+ | jobid=${PBS_JOBID%%.*} | ||
+ | |||
+ | # check how much time is left and set the " | ||
+ | checktime() { | ||
+ | if [ -x $qstat ] ; then | ||
+ | times=$(qstat -n1 $jobid | tail -n 1) | ||
+ | let tend=$(echo $times | awk ' | ||
+ | let tnow=$(echo $times | awk ' | ||
+ | let trem=$tend-$tnow | ||
+ | let tmin=$tnow/ | ||
+ | if [ $trem -ge $tmin ] ; then | ||
+ | moretime=" | ||
+ | else | ||
+ | moretime=" | ||
+ | fi | ||
+ | else | ||
+ | # cannot say => random guess | ||
+ | moretime=" | ||
+ | fi | ||
+ | } | ||
+ | |||
+ | # Execute a task as many times as possible. | ||
+ | let niter=0; | ||
+ | moretime=" | ||
+ | while [ " | ||
+ | # run your program here | ||
+ | ./ | ||
+ | let niter=$niter+1 | ||
+ | checktime | ||
+ | done | ||
+ | </ | ||
slurm.txt · Last modified: 2020/03/02 17:29 by admin