GPUs Basel 2018
Here we describe on how to use the GPUs provided for the Basel Workshop 2018. We go through each step by using a simple tutorial dataset and project as an example. You can use the same steps described on your dataset/project.
The GPUs we use are located on the high performance computing cluster of the University of Basel called sciCORE (https://scicore.unibas.ch) which uses the SLURM queuing system. A queuing system coordinates the access to the GPUs and is needed when there are many users using just a few GPUs.
The main idea is that we create an alignment project locally, move it to the cluster on sciCORE and then run it using a pre-installed Dynamo standalone version on sciCORE. To do that, we follow the following steps:
On your local Matlab session with Dynamo loaded:
- Create the tutorial project:
dtutorial myParticles -p myProject -M 128
We now have a tutorial dataset with 128 particles in the directory
myParticles and a tutorial alignment project
- Open the alignment project window:
and under computing environment select GPU (standalone) as an environment.
- Check and Unfold the project.
- Before moving the data to sciCORE we have to compress the project. In the project window go to Tools and then "create a tarball".
On your local Linux terminal:
- Copy the project data (particles) to sciCORE:
rsync -avuP myParticles USERNAME@login.bc2.unibas.ch:/scicore/home/PATH/dynamo_projects
- Copy the previously created tar file of the project to sciCORE:
rsync -avuP dTutorial.tar USERNAME@login.bc2.unibas.ch:/scicore/home/PATH/dynamo_projects
- Login to your sciCORE account:
ssh -Y USERNAME@login.scicore.unibas.ch
While logged in to your sciCORE account:
- Activate dynamo:
- Untar the Dynamo project:
dynamo dvuntar myProject.tar
- Create a blank SLURM submission script (text file) named submit_job.sh:
- Copy and adapt the following lines into the newly created script (note the difference between K80 and TitanX GPUs):
For using the K80 GPUs:
#!/bin/bash -l # #SBATCH --job-name=dTest #SBATCH --qos=30min #SBATCH --time=00:30:00 #SBATCH --mem=16G #SBATCH --nodes=1 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=1 #SBATCH --partition=k80 #SBATCH --gres=gpu:1 module load CUDA/7.5.18 source PATH/dynamo_activate_linux_shipped_MCR.sh cd PATH/dynamo_projects echo "dvput myProject -gpu_identifier_set $CUDA_VISIBLE_DEVICES" > dcommands.sh echo "dvunfold myProject" >> dcommands.sh dynamo dcommands.sh chmod u=rxw ./myProject.m ./myProject.m
For using the TitanX GPUs:
#!/bin/bash -l # #SBATCH --job-name=dTest #SBATCH --qos=emgpu #SBATCH --time=00:30:00 #SBATCH --mem=16G #SBATCH --nodes=1 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=1 #SBATCH --partition=titanx #SBATCH --gres=gpu:1 module load CUDA/7.5.18 source PATH/dynamo_activate_linux_shipped_MCR.sh cd PATH/dynamo_projects echo "dvput myProject -gpu_identifier_set $CUDA_VISIBLE_DEVICES" > dcommands.sh echo "dvunfold myProject" >> dcommands.sh dynamo dcommands.sh chmod u=rxw ./myProject.m ./myProject.m
- Note that depending on your project you might have to adapt the project name and the time requested (time=hh:mm:ss) in the script. If you are using the K80 and think that your job will run longer than 30 minutes, set the qos to 6hours and adapt the time to anything between 30 minutes and 6 hours.
- You can now run your alignment project by submitting the previously created script to SLURM with:
- With the following commands you can check the overall status of the submitted jobs:
Check your status in the queue:
squeue -u USERNAME
See all users in queue for the K80 GPU:
squeue -q 30min
See all users in queue for the TitanX GPU:
squeue -q empgu
To cancel the job type scancel followed by the job ID that was shown by the squeue command:
Some ways to check the last output:
ls -rtl tail -f slurm-45994509.out less slurm-45994509.out
To check the last average load the standalone Dynamo environment by typing
dynamo into the terminal and use the usual commands, e.g.:
ddb dTutorial:a -v