GPUs EMBO 2016

CSCS in Lugano is the Swiss Nacional Supercomputing Centre. CSCS kindly provides the EMBO course with 20 accounts. Each account should be able to submit jobs to a single node connected to a K20 GPU and four CPU cores.

Connecting with CSCS

First you need to connect to the gate node ela using your cscs credentials from the credentials handout.

ssh -Y course27@ela.cscs.ch

and then you can connect to the computing machine called daint, again you will be requested to type in your credentials.

course27@ela2:~> ssh -Y daint

Installing Dynamo

An installation is ready in the account course27. Just copy it onto your own account.

scp -r /users/course27/dynamo ~

Afterwards you will be able to activate Dynamo in your linux session by:

source /users/courseXX/dynamo/dynamo_activate_linux_shipped_MCR.sh

Please place your account number instead of the placeholder XX!

Using Dynamo

We are using a slightly older version of Dynamo on the supercomputer GPUs for compatibility reasons.

In a remote machine, things run differently: computing power is orders of magnitude higher, at the price of interfaces running slower and Matlab normally not being available. This makes the way you interact with Dynamo slightly more complicated. Commandline management of projects is often a more efficient way than using GUIs. We recommend to use two shells in the remote machine.

• One remote shell is used to open a Dynamo console.

In this shell, we can create and manage projects, inspect files, even show moderately complex graphics. We call this the console shell. In this console shell, we don't have don't need to have access to a GPU.

• Another the shell is used for number crunching, we call it a computing shell. This needs to "see" the GPU.

Creating a remote Dynamo console

• ssh to daint
• activate Dynamo in the shell:

source /users/courseXX/dynamo/dynamo_activate_linux_shipped_MCR.sh

• launch the console:

dynamo

Creating a remote Dynamo computing shell

• ssh to daint (in a different shell)
• request an interactive session:

salloc --gres=gpu:1

• Remark: during the course we should have a reservation for priority use of the nodes. In theory you should be able to use the syntax:

salloc --gres=gpu:1 --reservation=gpu --time=04:00:00

should allocate a session of uninterrumpted GPU usage for 4 hours.

• enable the CUDA environment:

module load cudatoolkit

• optional: check that the environment "sees" the gpu:

srun nvidia-smi

• activate Dynamo in the shell

source /users/courseXX/dynamo/dynamo_activate_linux_shipped_MCR.sh

Exercises

Exercise 1: Creating tutorial projects

We can use the system terminal as an equivalent of the Matlab terminal using the Dynamo standalone. This is an example on how to use it to create a phantom project like the one we did yesterday.

• open a Dynamo console by typing:

dynamo

in a linux shell (you'll need to source Dynamo activation script on that shell beforehand).

• create a tutorial project. For this, type inside the Dynamo console:

dtutorial myTest -p ptest -M 128

• tune the project to work in a GPU

dvput ptest -destination matlab_gpu

• unfold the project

dvunfold ptest inside the Dynamo console

• run the project with srun in a computing shell

srun dynamo_execute_project ptest in a terminal shell, i.e., not inside the Dynamo console

It can take a long time for starting the project, as it needs to awaken the MCR libraries. After this initial latency, there is no additional overhead for the use of the standalone.

srun its a syntax required by CSCS: all commands that will talk to a GPU need to be preceded by this routing command.

the syntax dynamo_execute_project <projectname> is non-standard just for this exercise.

• while the project is running in computing shell, you can check its progress in the console shell by:

dvstatus ptest

• when it finishes, the averages can be also accessed programmatically with the database tool. For instance, to access the last computed average and view it with dview, type:

ddb ptest:a -v

Exercise 2: Transferring projects

In this example, we show how to transfer a project from a local machine into the remote system, by Dynamo-tarring a project in a local machine, copying it into a remote machine and untarring it there.

On the local machine

1. tar your project in Dynamo (in Dynamo wizard >> Tools >> Create a tarball
2. rsync -avr my_project.tar stud##@ela.cscs.ch:~/
3. Also rsync your data to CSCS
4. Untar your Dynamo project

You will need the Dynamo terminal for this:

dynamo &

dvuntar myProject

On CSCS,

1. type
   salloc --gres=gpu:1
   to get a node with a gpu. It can take some time till the system allocates you a node. You can allocate up to two nodes.
   you can check the GPU on your node by:
   srun nvidia-smi
2. type
   source ~/dynamo/dynamo_activate_linux_shipped_MCR.sh
   to activate Dynamo in your shell.
3. open Dynamo with dynamo &
4. open your project, and re-unfold it (make sure standalone GPU is selected and make sure your data is in the same relative location as on the local machine)

   Note

   if the graphical interface is too slow, you can use the command line instead:

   open a Dynamo console in your shell with dynamo

   dvput my_project -data dataFolderInRemoteHost

   dvput my_project -destination matlab_gpu

   dvunfold my_project

5. run your alignment by typing srun dynamo_execute_project my_project

Note about performance You will notice that the project stops at several points during execution. Those are the points where the project accesses the MCR libraries. This overhead is a constant, and is a very small fraction of the computing time for a real project with thousands of particles.

We are using an old Dynamo version. Modern Dynamo versions don't access the MCR library several times.