Skip to content

Batch processing guide (HPC)

Computing servers

Now we already know where to save our code (our home).

The remaining question is where to really run our experiments?

If everybody run their processing-intensive experiments, that will probably use a lot of RAM memory, everything in the *e01 server, then these servers will probably collapse, or run too slowly form normal coding.

For this reason we have a computing service, to be able to run a lot of experiments using all our computing servers, but keeping our access servers free to be used to code or other non-processing-intensive tasks.

Monitoring the computing service

To see the resources information of the computing service and partitions, you can use sinfo:

>> sinfo 

To see the current status of the computing service just type squeue or sview:

>> squeue 
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
            240123  standard   python asalvado  R 2-01:59:39      1 c5
            264825  standard matrix_c etacchin  R    2:01:29      1 c5

In this example you can see that the user asalvado is running (R) a python experiment for more than 2 days, and etacchin some matrices-related one for more than 2 hours, both in c5 node.

If you run sview you can see more details, like the RAM or the number of cores they are using, etc...

Anyway, we are lucky\\ There is only a couple of users working, so there is free space for us to also use it, let's do it!

Submitting a job

Do you know the sleep command? Well, it just do nothing for the seconds that you specify as a parameter... quite simple, right?

As any other command, if you simply type it in the shell, it will run the host you are logged in (gpie01 in our case):

>> sleep 10 &
[1] 30329
albert.gil@gpie01:~>> squeue 
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
            240123  standard   python asalvado  R 2-01:59:49      1 c5
            264825  standard matrix_c etacchin  R    2:01:39      1 c5

Please note that the final & of the sleep 10 & command is just to recover the terminal after the call, to not wait until it finish (more info about running shell commands).

Then, after calling sleep 10, we can see that nothing changed in the computing service... ok. But if we simply type srun before it, then it will run in the computing service:

albert.gil@gpie01:~>> srun sleep 10&
[1] 31262
albert.gil@gpie01:~>> squeue 
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
            240123  standard   python asalvado  R 2-02:00:22      1 c5
            264825  standard matrix_c etacchin  R    2:02:12      1 c5
            264836  standard    sleep     agil  R       0:03      1 c5

In the last line of the squeue command you can see that our sleep is now running, not it gpie01, but in the c5 node.

Submitting a job to a partition queue

The computing service is divided into several partition queues, like in a supermarket. The gpi.compute one allows to run your command with a big time limit (24h), but you will probably wait in the cue until there are resources available for you, while the gpi.develop one allows only to run jobs that take less than 30 minutes, but you won't wait in the queue so long, ideally you will have access immediately. To use it:

albert.gil@gpie01:~>> srun -p gpi.develop sleep 10&

In any case, if you need to compute something, please submit it into the system and wait until it is computed, do not cancel your own jobs because they are waiting. Cancelling jobs that you actually want to compute is bad for several reasons:

  • Of course, it won't be computed until you submit it again...
  • The more time that your job is in the queue the more priority it gains to be the next to be run.
  • The length of the queue allows us to determine the actual computing needs of the group, that is, your actual computing needs.
    • If you "wait" to submit (without any jobs in the queue) instead of waiting in the queue, for the system you are not actually waiting.

Cancelling a job

To cancel your running job you have to get its jobid by running squeue, and the just:

>> scancel $JOBID

Asking for computing resources

CPUs

To ask for 2 CPUs:

>> srun -c 2 ./myapp

RAM

To ask for 4G of RAM:

>> srun --mem 4G ./myapp

Please note that, if you get the message srun: error: task: Killed, it is probably because your app is using more RAM that your reserved. Just ask for more RAM, or try to reduce the amount of RAM that you need.

GPUs and their RAM

GPUs are part of the Generic Resources (gres), so to ask for 1 GPU:

>> srun --gres=gpu:1 ./myapp

If you want an GPU with at least 6GB of RAM:

>> srun --gres=gpu:1,gpumem:6G ./myapp

Usually you need to check how much GPU RAM is your job actually using. The nvidia-smi is the right command to use, but to integrate it into our comuting service, you should use:

>> srun-monitor-gpu $JOBID

Where $JOBID is the jobid of your running job. You can see it in the first lines when your job starts, or in the output of the squeue command.

Graphical Interfaces

If you job does any kind of graphical visualization (opens any window), then you need to pass an extra parameter --x11:

>> srun --x11 ./myapp

Submit jobs with a batch file

You also can create an automated script, in this case you ask for a GPU. Look at this example myscript.sh

#!/bin/bash
#SBATCH -p veu             # Partition to submit to
#SBATCH --mem=1G      # Max CPU Memory
#SBATCH --gres=gpu:1
python myprogram.py
>> sbatch myscript.sh

Job arrays

If we plan to run the same experiment hundreds or thousands of times but just changing some parameters, the Job Array is our friend.

The best way to understand them is following an example.
Imagine tha we want to run this three commands:

>> srun  myapp --param 10
>> srun  myapp --param 15
>> srun  myapp --param 20

In this simple use case we could just do it like that: three srun commands.
But if we want to test myapp with hundreds or thousands of parameters, do it manually is not an option; and job arrays are here for the rescue!

To achieve this three commands using Job Arrays we always need to create a shell script that we can call worker.sh. A first approach to solve the problem could be a worker.sh like this:

#!/bin/bash
myapp --param $SLURM_ARRAY_TASK_ID

And we should launch it with:

>> sbatch --array=10,15,20 worker.sh

Here we should note several things:

  • The sbatch command is almost like srun, but for shell scripts
  • The worker.sh script is srunned all the times requested in the --array param of the sbatch command.
  • The values and ranges passed to the --array are converted to the $SLURM_ARRAY_TASK_ID variable in each execution
  • By default the standard output and error are saved in files like slurm-jobid-taskid.out in the current directory

For example, let's try our first hello job array with a simple worker that just prints out a hello world message, and waits a little bit for convenience:

imatge@nx2:~>> cat worker.sh 
#!/bin/bash
echo "Hello job array with parameter: " $SLURM_ARRAY_TASK_ID
sleep 10 # to be able to run squeue

We can launch 5 jobs in an array with:

>> sbatch --array=0-5 worker.sh
Submitted batch job 43702

If we monitor the queue we see our 5 executions:

>> squeue 
               JOBID PARTITION      NAME     USER ST       TIME CPUS MIN_MEM GRES            NODELIST(REASON)
             43702_0  standard worker.sh   imatge  R       0:02    1    256M (null)          v5
             43702_1  standard worker.sh   imatge  R       0:02    1    256M (null)          c4
             43702_2  standard worker.sh   imatge  R       0:02    1    256M (null)          c4
             43702_3  standard worker.sh   imatge  R       0:02    1    256M (null)          c4
             43702_4  standard worker.sh   imatge  R       0:02    1    256M (null)          c4
             43702_5  standard worker.sh   imatge  R       0:02    1    256M (null)          c4

Once they are done, we can see the output files in the current directory following the slurm-jobid-taskid names:

>> ll slurm-43702_*
-rw-r--r-- 1 imatge imatge 35 Nov  4 19:21 slurm-43702_0.out
-rw-r--r-- 1 imatge imatge 35 Nov  4 19:20 slurm-43702_1.out
-rw-r--r-- 1 imatge imatge 35 Nov  4 19:20 slurm-43702_2.out
-rw-r--r-- 1 imatge imatge 35 Nov  4 19:20 slurm-43702_3.out
-rw-r--r-- 1 imatge imatge 35 Nov  4 19:20 slurm-43702_4.out
-rw-r--r-- 1 imatge imatge 35 Nov  4 19:20 slurm-43702_5.out

And we can check the expected output contents:

>> cat slurm-43702_*
Hello job array with parameter:  0
Hello job array with parameter:  1
Hello job array with parameter:  2
Hello job array with parameter:  3
Hello job array with parameter:  4
Hello job array with parameter:  5

Note that you can combine ranges and values in the --array parameter of the sbatch command like this:

>> sbatch --array=1-3,5-7,100 worker.sh
Submitted batch job 43710

>> cat slurm-43710_*
Hello job array with parameter:  100
Hello job array with parameter:  1
Hello job array with parameter:  2
Hello job array with parameter:  3
Hello job array with parameter:  5
Hello job array with parameter:  6
Hello job array with parameter:  7

This range and values flexibility could very useful when for example we run thousands of workers, but just some of the failed and you want to rerun only those.

Note also that we can also use a Python script (or any other script language) as a worker, like this:

>> cat worker.py 
#!/usr/bin/python
import os
print "Hello job array with Python, parameter: ", os.environ['SLURM_ARRAY_TASK_ID']

>> sbatch --array=1-3 worker.py
Submitted batch job 43720

>> cat slurm-43720_*
Hello job array with Python, parameter:  1
Hello job array with Python, parameter:  2
Hello job array with Python, parameter:  3

We can use arrays or dictoinaries in our script to handle multple parameters of our worker script:

>> cat worker.sh
#!/bin/bash

param1[0]="Bob"; param2[0]="Monday"
param1[1]="Sam"; param2[1]="Monday"
param1[2]="Bob"; param2[2]="Tuesday"
param1[3]="Sam"; param2[3]="Tuesday"

echo "My friend" ${param1[$SLURM_ARRAY_TASK_ID]} "will come on" ${param2[$SLURM_ARRAY_TASK_ID]}

>> sbatch --array=0-3 worker.sh
Submitted batch job 43734

>> cat slurm-43734_*
My friend Bob will come on Monday
My friend Sam will come on Monday
My friend Bob will come on Tuesday
My friend Sam will come on Tuesday

To handle parameters we can alse use external files and use the SLURM_ARRAY_TASK_ID as the line nuber to read the desired parameters, like this:

>> cat param1.txt 
Bob
Sam
Bob
Sam
>> cat param2.txt 
Monday
Monday
Tuesday
Tuesday

>> cat worker.sh
#!/bin/bash
param1=`sed "${SLURM_ARRAY_TASK_ID}q;d" param1.txt`
param2=`sed "${SLURM_ARRAY_TASK_ID}q;d" param2.txt`
echo "My friend" $param1 "will come on" $param2

>> sbatch --array=1-4 worker.sh
Submitted batch job 43740

>> cat slurm-43740_*
My friend Bob will come on Monday
My friend Sam will come on Monday
My friend Bob will come on Tuesday
My friend Sam will come on Tuesday

Please note that in this case we have to use the range 1-4 instead of the range 0-3 that we used when we save the parameters in arrays instead of files.