Difference between revisions of "GPUs Basel 2018"

From Dynamo
Jump to navigation Jump to search
 
(9 intermediate revisions by the same user not shown)
Line 1: Line 1:
Here we describe on how to use the GPUs provided for the Basel Workshop 2018. We go through each step by using a simple tutorial dataset/project as an example. You can use the same steps on your dataset/project of choice.
+
Here we describe how to use the GPUs provided for the Basel Workshop 2018. We go through each step and use a simple tutorial dataset and project as an example. You can use the same steps described here on your own project.
  
The GPUs we use are located on the high performance computing cluster of the University of Basel called sciCORE (https://scicore.unibas.ch) which uses the SLURM queuing system. A queuing system coordinates the access to the GPUs and is needed when there are many users using just a few GPUs.
+
The GPUs we use are located on the high performance computing cluster of the University of Basel called sciCORE (https://scicore.unibas.ch) which uses the SLURM queuing system. A queuing system coordinates the access to the GPUs and is needed when there are many users using a limited amount of GPUs. You have been given the credentials needed to log in on the cluster at the beginning of the workshop.
  
We will create an alignment project locally, move it to sciCORE and run it there using a pre-installed Dynamo standalone version.
+
The main idea is that we create an alignment project locally, move it to the cluster on sciCORE and then run it using a pre-installed Dynamo standalone version on sciCORE. How you can do that is described in the following steps.
  
  
On your local Matlab session with dynamo loaded:
+
'''On your local Matlab session with Dynamo loaded:'''
1) Create the tutorial project: dtutorial myParticles -p myProject -M 128
 
2) Open the alignment project window with dcp myProject and in computing environment select gpu as computing environment. The rest remains default.
 
3) Check and Unfold the project
 
4) Before moving the data to sciCORE we have to compress the project: in dcp gui go to tools and then create tarball
 
  
On local linux terminal:
+
* Create a tutorial project with Dynamo:
7) copy project data (particles) to sciCORE with following command:
+
<tt>dtutorial myParticles -p myProject -M 128</tt>
rsync -avuP myParticles USERNAME@login.bc2.unibas.ch:/scicore/home/.../dynamo_projects
+
We now have a tutorial dataset with 128 particles in the directory <code>myParticles</code> and a tutorial alignment project <code>myProject</code>.
8) copy tar of project to scicore:
 
rsync -avuP dTutorial.tar scaramuz@login.bc2.unibas.ch:/scicore/home/.../dynamo_projects
 
9) login to scicore:
 
ssh -Y USERNAME@login.scicore.unibas.ch
 
  
On scicore:
+
* Open the alignment project window:
13) activate dynamo:
+
<tt>dcp myProject</tt>
source PATH/dynamo_activate_linux_shipped_MCR.sh
+
and under ''computing environment'' select ''GPU (standalone)'' as an environment.
14) untar dynamo project:
 
dynamo dvuntar myProject.tar
 
15) create SLURM submission script "submit_job.sh":
 
  
#!/bin/bash -l
+
* Check and unfold the project.
 +
 
 +
* Before moving the data to sciCORE we have to compress the project. In the project window go to ''Tools'' and then ''create a tarball''.
 +
 
 +
 
 +
'''On your local Linux terminal:'''
 +
 
 +
* Open a local Linux terminal and navigate to the directory where you just created the tutorial dataset and project. Copy the project data (particles) to sciCORE:
 +
<tt>rsync -avuP myParticles USERNAME@login.scicore.unibas.ch:/scicore/home/s-gpu-course/USERNAME/dynamo_projects</tt>
 +
 
 +
* Copy the previously created tar file of the project to sciCORE:
 +
<tt>rsync -avuP myProject.tar USERNAME@login.scicore.unibas.ch:/scicore/home/s-gpu-course/USERNAME/dynamo_projects</tt>
 +
 
 +
* Login to your sciCORE account:
 +
<tt>ssh -Y USERNAME@login.scicore.unibas.ch</tt>
 +
If asked to continue type "yes". Use the provided password.
 +
 
 +
 
 +
'''While logged in to your sciCORE account:'''
 +
 
 +
* While logged in to your sciCORE account, activate dynamo:
 +
<tt>source /scicore/home/s-gpu-course/GROUP/dynamo_activate_linux_shipped_MCR.sh</tt>
 +
 
 +
* Go to the location where you copied the data:
 +
<tt>cd dynamo_projects </tt>
 +
 
 +
* Untar the Dynamo project:
 +
<tt>dynamo </tt>
 +
<tt>dvuntar myProject.tar </tt>
 +
<tt>exit </tt>
 +
 
 +
* Create a blank SLURM submission script (text file) named ''submit_job.sh'':
 +
<tt>nano submit_job.sh</tt>
 +
 
 +
* Copy (and adapt) the following lines into the newly created script. Depending on your project you might have to adapt the project name and the time requested (''time=hh:mm:ss'') in the script. If your job will run longer than 30 minutes, set the ''qos'' to ''6hours'' and adapt the time to anything between 30 minutes and 6 hours:
 +
 
 +
<nowiki>#!/bin/bash -l
 
#
 
#
 
#SBATCH --job-name=dTest
 
#SBATCH --job-name=dTest
#SBATCH --qos=30min       (for titanX: emgpu)
+
#SBATCH --qos=30min
#SBATCH --time=00:60:00 (adapt time)
+
#SBATCH --time=00:30:00
 
#SBATCH --mem=16G
 
#SBATCH --mem=16G
 
#SBATCH --nodes=1
 
#SBATCH --nodes=1
 
#SBATCH --ntasks-per-node=1
 
#SBATCH --ntasks-per-node=1
 
#SBATCH --cpus-per-task=1
 
#SBATCH --cpus-per-task=1
#SBATCH --partition=k80  (for titanX: titanx)
+
#SBATCH --partition=pascal
 
#SBATCH --gres=gpu:1
 
#SBATCH --gres=gpu:1
 +
#SBATCH --reservation=dynamo
 
module load CUDA/7.5.18
 
module load CUDA/7.5.18
source PATH/dynamo_activate_linux_shipped_MCR.sh
+
source /scicore/home/s-gpu-course/GROUP/dynamo_activate_linux_shipped_MCR.sh
cd PATH/dynamo_projects
+
cd $HOME/dynamo_projects
 
echo "dvput myProject -gpu_identifier_set $CUDA_VISIBLE_DEVICES" > dcommands.sh
 
echo "dvput myProject -gpu_identifier_set $CUDA_VISIBLE_DEVICES" > dcommands.sh
 
echo "dvunfold myProject" >> dcommands.sh
 
echo "dvunfold myProject" >> dcommands.sh
 
dynamo dcommands.sh
 
dynamo dcommands.sh
chmod u=rxw ./myProject.m
+
chmod u=rxw ./myProject.exe
./myProject.m
+
./myProject.exe</nowiki>
 +
 
 +
 
 +
* You can now run your alignment project by submitting the previously created script to SLURM with:
 +
<tt>sbatch submit_job.sh</tt>
  
16) launch job on slurm with:
+
* With the following commands you can check the overall status of the submitted jobs:
sbatch submit_job.sh
 
  
17) check queue:
+
Check your status in the queue:
squeue -u USERNAME
+
<tt>squeue -u USERNAME</tt>
  
see all users in queue:
+
See all users in the queue:
squeue -q 30min (for titanX: squeue -q empgu)
+
<tt>squeue -q 30min</tt>
  
  
18) cancel job:
+
To cancel the job type ''scancel'' followed by the job ID that was shown by the squeue command:
scancel ????? (job id given from squeue command)
+
<tt>scancel my_job_id</tt>
  
19) check last output:
+
Some ways to check the last output:
ls -rtl
+
<tt>ls -rtl</tt>
tail -f slurm-45994509.out
+
<tt>tail -f slurm-45994509.out</tt>
less slurm-45994509.out
+
<tt>less slurm-45994509.out</tt>
  
20) check last average
+
To check the last average load the standalone Dynamo environment by typing <code>dynamo</code> into the terminal and use the usual commands, e.g.:
dynamo
+
<tt>ddb myProject:a -v</tt>
ddb dTutorial:a -v
 

Latest revision as of 14:22, 27 August 2018

Here we describe how to use the GPUs provided for the Basel Workshop 2018. We go through each step and use a simple tutorial dataset and project as an example. You can use the same steps described here on your own project.

The GPUs we use are located on the high performance computing cluster of the University of Basel called sciCORE (https://scicore.unibas.ch) which uses the SLURM queuing system. A queuing system coordinates the access to the GPUs and is needed when there are many users using a limited amount of GPUs. You have been given the credentials needed to log in on the cluster at the beginning of the workshop.

The main idea is that we create an alignment project locally, move it to the cluster on sciCORE and then run it using a pre-installed Dynamo standalone version on sciCORE. How you can do that is described in the following steps.


On your local Matlab session with Dynamo loaded:

  • Create a tutorial project with Dynamo:
dtutorial myParticles -p myProject -M 128

We now have a tutorial dataset with 128 particles in the directory myParticles and a tutorial alignment project myProject.

  • Open the alignment project window:
dcp myProject

and under computing environment select GPU (standalone) as an environment.

  • Check and unfold the project.
  • Before moving the data to sciCORE we have to compress the project. In the project window go to Tools and then create a tarball.


On your local Linux terminal:

  • Open a local Linux terminal and navigate to the directory where you just created the tutorial dataset and project. Copy the project data (particles) to sciCORE:
rsync -avuP myParticles USERNAME@login.scicore.unibas.ch:/scicore/home/s-gpu-course/USERNAME/dynamo_projects
  • Copy the previously created tar file of the project to sciCORE:
rsync -avuP myProject.tar USERNAME@login.scicore.unibas.ch:/scicore/home/s-gpu-course/USERNAME/dynamo_projects
  • Login to your sciCORE account:
ssh -Y USERNAME@login.scicore.unibas.ch

If asked to continue type "yes". Use the provided password.


While logged in to your sciCORE account:

  • While logged in to your sciCORE account, activate dynamo:
source /scicore/home/s-gpu-course/GROUP/dynamo_activate_linux_shipped_MCR.sh
  • Go to the location where you copied the data:
cd dynamo_projects 
  • Untar the Dynamo project:
dynamo 
dvuntar myProject.tar 
exit 
  • Create a blank SLURM submission script (text file) named submit_job.sh:
nano submit_job.sh
  • Copy (and adapt) the following lines into the newly created script. Depending on your project you might have to adapt the project name and the time requested (time=hh:mm:ss) in the script. If your job will run longer than 30 minutes, set the qos to 6hours and adapt the time to anything between 30 minutes and 6 hours:
#!/bin/bash -l
#
#SBATCH --job-name=dTest
#SBATCH --qos=30min
#SBATCH --time=00:30:00
#SBATCH --mem=16G
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=1
#SBATCH --partition=pascal
#SBATCH --gres=gpu:1
#SBATCH --reservation=dynamo
module load CUDA/7.5.18
source /scicore/home/s-gpu-course/GROUP/dynamo_activate_linux_shipped_MCR.sh
cd $HOME/dynamo_projects
echo "dvput myProject -gpu_identifier_set $CUDA_VISIBLE_DEVICES" > dcommands.sh
echo "dvunfold myProject" >> dcommands.sh
dynamo dcommands.sh
chmod u=rxw ./myProject.exe
./myProject.exe


  • You can now run your alignment project by submitting the previously created script to SLURM with:
sbatch submit_job.sh
  • With the following commands you can check the overall status of the submitted jobs:

Check your status in the queue:

squeue -u USERNAME

See all users in the queue:

squeue -q 30min


To cancel the job type scancel followed by the job ID that was shown by the squeue command:

scancel my_job_id

Some ways to check the last output:

ls -rtl
tail -f slurm-45994509.out
less slurm-45994509.out

To check the last average load the standalone Dynamo environment by typing dynamo into the terminal and use the usual commands, e.g.:

ddb myProject:a -v