Difference between revisions of "Starters guide (workshop version)"

From Dynamo
Jump to navigation Jump to search
 
(46 intermediate revisions by the same user not shown)
Line 9: Line 9:
 
[[File:StartersGuideFigure1.png|thumb|center|600px| Figure 1: creating a set of synthetic tomograms with ''Dynamo'' Catalogue Manager]]
 
[[File:StartersGuideFigure1.png|thumb|center|600px| Figure 1: creating a set of synthetic tomograms with ''Dynamo'' Catalogue Manager]]
  
The tomograms are situated in the directory <tt>testCatalogue</tt> along with the ground truth for particle locations and orientations. Tomograms contain small amount of noise and a missing wedge associated to rotation around Y-axis. The information about location of the particles in the tomograms is stored in a catalogue called <tt>testCatalogue_withmodels</tt>. Existing catalogues from the current folder may be recalled with ''Catalogue -> look for local catalogues'' in the catalogue manager window.
+
You can see the list of your tomograms and their metadata in a table in the bottom of your catalogue manager. When workin on your own projects, you can add tomograms to the catalogue by ''Catalogue -> Browse for new volume''. The tomograms are situated in the directory <tt>testCatalogue</tt>. They contain a small amount of noise and a missing wedge associated to rotation around Y-axis. We added the information about location of the particles in the tomograms in an additional catalogue called <tt>testCatalogue_withmodels</tt>. If needed, it can be opened from the current folder with ''Catalogue -> look for local catalogues'' in the catalogue manager window.
  
 
==Viewing tomograms==
 
==Viewing tomograms==
You can see the list of your tomograms and metadata in a table in the bottom of your [[dcm | catalogue manager window]]. You can add tomograms to the catalogue by ''Catalogue -> Browse for new volume''. You can get an overview of your tomograms in the catalogue by ''View all volumes -> create thumbnails'' and ''View all volumes -> show thumbnail gallery''. All these functionalities have their corresponding command line versions, accessible through the <tt>dcm</tt> command (short for <tt>dynamo_catalogue_manager</tt>), whose options and syntax can be consulted through <tt>>> doc dcm</tt>.
+
Select the first tomogram in the list and go to ''View volume>Full tomogram file in tomoslice''.  
''Dynamo'' has several viewers designed for various purposes. Select the first tomogram in the list with a secondary mouse click and go to ''View volume>Full tomogram file in tomoslice'' .  
 
  
 
[[ File:StartersSelectTomoSliceInCatalogue.png |thumb| center |600px| Figure 2: Selecting a volume in the catalogue manager to be inspected in <tt>dpreview</tt>]]
 
[[ File:StartersSelectTomoSliceInCatalogue.png |thumb| center |600px| Figure 2: Selecting a volume in the catalogue manager to be inspected in <tt>dpreview</tt>]]
  
The volume browser [[dtmslice | tomoslicer]] loads the entire tomograms into memory and allow making annotations to the regions of interest. The tomograms in this tutorial are small, and you can load them directly into memory. For real life tomograms, bear in mind that you will need to [[Prebinned tomograms| prebin]] the files before loading them into memory (or alternatively, load fragments of them using <tt>dpreview</tt>)
+
The volume browser tomoslicer loads the entire tomograms into memory and allows making annotations to the regions of interest. The tomograms in this tutorial are small, and you can load them directly into memory. For real life tomograms, bear in mind that you will need to [[Prebinned tomograms| prebin]] the files before loading them into memory. Tomoslice has a simple set of controls and is suitable for visualization tasks that require oblique sections through the tomogram. It uses the same tool as other ''Dynamo'' browsers to keep track of your annotations: a pool of models. You might need to adjust the contrast (blue arrow). To move through the tomogram slices, you can either use the mouse wheel, click and drag the tomogram slice up and down (orange arrows), or move the position control left and right (orange box).
  
Tomoslice has a simple set of controls, and is  suitable for visualization tasks that require oblique sections through the tomogram. It uses the same tool as other ''Dynamo'' browsers to keep track of your annotations: a pool of models.
 
  
Go to ''View volume -> load full tomogram into Tomoslice''. On the left control panels Scene turn off the 3d view radiobutton. You can navigate the z-height of the tomogram with a mouse wheel or with a slider in the tomoslice menu (1 to 256 in this case); you can also change the slice thickness from the default 10 pixels. Play with the controls and explore the tomogram. Turn 3d view back on.
+
[[ File:StartersTomoSlice.png |thumb| center |600px|]]
  
==Picking particles in tomograms==
+
==Picking and extracting particles in tomograms==
Coordinates of picked particles are represented by data types called models. In the tomoslicer window go to ''Model Pool -> Create new model in pool (choose type) -> General''. This is the simplest type of [[model]] where each clicked/model point corresponds to a single isolated particle. Now you can navigate up and down the tomogram, center the mouse on the particle and click “C” to add a new model point. (Note the ''Help -> Shortcut'' options that lists the different the actions of the different keystrokes). Backspace button deletes the last clicked point.
+
Coordinates of picked particles are represented by data types called models. In the tomoslicer window go to ''Model Pool -> Create new model in pool (choose type) -> General''. This is the simplest type of [[model]] where each clicked/model point corresponds to a single isolated particle. Now you can navigate up and down the tomogram, place the mouse on the center of a particle and press the [c] key on your keyboard to add a new model point. (Note the ''Help -> all hot keys'' options that lists the different the actions of the different keystrokes). Backspace button deletes the last clicked point (you can also use the right-click to delete single points).
  
[[File:StartersGuideFigure4.png|thumb|center|600px| Figure 4. Generating a new model]]
+
[[File:StartersGuideFigure4.png|thumb|center|600px| Generating a new model]]
  
After you are done clicking a dozen of particles go to ''Active model -> Edit active model''. This is a summary of the model with control elements for geometric transformations. Models of type general don’t allow any geometrical transforms, all you need to do here is to convert the clicked model points into “table points” that will be used for cropping. This is done by clicking the table-looking command icon (4th from the right in the top left part of the Editor of model parameters window). Alternatively, you could just click on ''Active model > Update Crop points'' in the tomoslice window.
+
After you are done clicking about 10 particles click on ''Active model > Update Crop points'' in the tomoslice window.
 

 

[[File:StartersGuideFigure5.png|thumb|center|600px| Figure 5. Updating crop points and extracting particles from the model editor window]]
 
  
The model editor also gives you the possibility of changing the name of the model. You may want to give the models meaningful identifiers before storing them into disk. Also, you can open a GUI for extracting your particles (first icon from the right) to extract particles from only this tomogram, however we will do it later for all 3 tomograms. Make sure that the number of clicked points equals to the number of table points in the Information field and close the Editor window. Importantly, save the model into the catalogue by ''Active model -> Save active model into catalogue (disk)'' and close the slicer window. Pick particles for tomograms 2 and 3. In total, there are around 40 particles in the generated dataset.
+
[[File:StartersGuideUpdate.png|thumb|center|600px|]]
  
[[File:StartersGuideFigure6.png|thumb|center|600px| Figure 6. Remember to save your work: use <tt>Update Crop</tt> to create table points, and then save the model into the catalogue.]]
+
Save the model into the catalogue by ''Active model -> Save active model into catalogue (disk)'' and close the slicer window.

+
 
Important: when you open a new tomogram, make sure that you delete the pool of models from memory (model pool > clear current pool from memory). This will have no effect onto the models stored in disk, and it is necessary in order to ensure that you are not mixing models from different tomograms.
+
[[File:StartersGuideFigure6.png|thumb|center|600px| ]]
 +
 
 +
Pick particles for tomograms 2 and 3. In total, there are around 30-40 particles in the generated dataset. When you open a new tomogram, make sure that you delete the pool of models from memory when asked. This will have no effect onto the models stored in disk, and it is necessary in order to ensure that you are not mixing models from different tomograms.
  
 
==An alternate visualization: orthogonal projections==
 
==An alternate visualization: orthogonal projections==
  
Clicking on <tt>Projection</tt> you will get a screen where the x-y projection of the tomogram is shown. Use the secondary click on it to launch the orthogonal views of x-z and y-z planes that traverse that point. Clicking on those views determines a 3D point.
+
Clicking in the menu on ''Projection -> project full shown fragment along z'' (still in dtmslice) and you will get a screen where the x-y projection of the tomogram is shown. Use the secondary click on it to launch the orthogonal views of x-z and y-z planes that traverse that point. These views can also be used to click particles, in case the standard view is not sufficient.
  
 
{|style="margin: 0 auto;"
 
{|style="margin: 0 auto;"
Line 49: Line 48:
  
 
=Extracting particles from tomograms=
 
=Extracting particles from tomograms=
In the catalogue manager window select the rows for tomograms from which you want to extract particles from. You can either select them one by one with a mouse click holding a CTRL button or click the “Select all” button.
+
In the catalogue manager window select the rows for tomograms from which you want to extract particles from. You can either select them one by one with a mouse click and holding the [ctrl] key or by clicking the ''Select all'' button. Then, go to ''Crop Particles -> Open Volume List Manager''.

+
 
[[File:StartersGuideFigure8.png|thumb|center|600px| Figure 8. Extracting particles with DCM and selecting the models for particle extraction.]]
+
[[File:StartersGuideopenVolumeList.png|thumb|center|600px|]]
  
Go to ''Crop Particles -> Open Volume List Manager'', all the models in the catalogue listed in the bottom of the window. Pick (by checking boxes) the models of type general – the ones that you have just clicked. Increase sidelength to 48-64 pixels. Vll-list is the format in which Dynamo stores directives for particle extraction; you can modify it with your own scripts if you have a lot of tomograms. Now click ''Create list'' and ''Crop particle''. Change output directory name to i.e. “particles” and click “start cropping”.
+
A new window opens with all the models in the catalogue listed in the bottom. Pick (by checking boxes) the models of type ''general'' that you have just clicked. Click ''Create list'' and then ''Crop particle''.
  
 +
[[File:StartersGuideSelectModels.png|thumb|center|600px|]]
  
[[File:StartersGuideFigure9.png|thumb|center|600px| Figure 9. Cropping particles from the set of selected models.]]
+
In the new window, change the data name to something meaningful, such as ''thermosomeParticles'', change the sidelength from 32 to 48 and click start cropping.
  
After cropping is done explore your cropped particles by clicking ddbrowse under the output folder name. <tt>ddbrowse</tt> (Dynamo Data Browse) is a lightweight browser for particle visualization, just click “show” and make sure your particles are well centered and fit the box. If not - use pre-existing models to generate the vll-file and re-extract the particles or increase the sidelength. Please note that ''Dynamo'' generated a particles/ folder with sub-volumes formatted as particle_000XX.em.
+
[[File:StartersGuideCrop.png|thumb|center|600px|]]
 +
 
 +
After cropping is done, explore your cropped particles by clicking ''ddbrowse'' (in the window above). A new window opens where you can simply click on ''show''.
 +
 
 +
[[File:StartersGuideShow.png|thumb|center|500px|]]
 +
 
 +
You see a 2D representation (projections) of all cropped 3D subtomograms. Make sure your particles are well centered and fit the box.
 +
 
 +
[[File:StartersGuideParticles.png|thumb|center|500px|]]
 +
 
 +
We also introduce the concept of the ''dynamo'' table: The information about each particle is stored in tables. Each particle has an entry in the table that contains shifts and rotations to describe its orientation (by default these values are initialized as zeroes). It also contains the particle ID (tag), the orientation of the missing wedge and others. For full info type the command <tt>dthelp</tt>. The ''Dynamo'' catalogue generates an initial table during particle extraction. Its location is in <tt>particles/crop.tbl </tt>.  
  
[[File:StartersGuideParticlesUnaligned.png|thumb|center|400px| Particles without alignment (<tt>table</tt> is switched off).]]
 

 
 
=Subtomogram alignment and averaging=
 
=Subtomogram alignment and averaging=
  
==Initial model generation==
+
==Initial reference generation==
If you don’t have an initial model it is easy to generate it by manually aligning several particles and summing them up. ''Dynamo'' stores information about the particles in dynamo tables: for each particle a table contains shifts and rotations to bring the particle to the average (by default these values are initialized as 0s), particle ID (tag), orientation of the missing wedge and others. For full info type <tt>>> dthelp</tt> in Matlab. ''Dynamo'' catalogue generates an initial table during particle extraction, its location is <tt>particles/crop.tbl </tt>
+
We want to align the extracted tomograms to a common reference. For that, we need an initial reference (which will later be refined during the iterative alignment). Here, we create such an initial reference by manually align a few particles and average them. We use the command <tt>dgallery</tt> to generate initial orientations for some of our particles that will be used to generate the initial reference. Close all open ''Dynamo'' windows and type into the ''Dynamo'' command line:
  
=== Loading particle in <tt>dgallery</tt> ===
+
<tt>dgallery('data','thermosomeParticles.Data');</tt>
  
For initial model generation we will use dynamo_gallery to generate initial orientations for some of our particles. Type
+
The gallery opens. Click on ''load'' to load all aprticles in memory, move the ''shown'' bar to display the particles and use the x-, y- and z-buttons to see different views:
  
<tt>>> dgallery(‘data’, ‘exampleData.Data’); </tt>
+
[[File:StartersGuideGallery.png|thumb|center|600px|]]
  
assuming you named <tt>exampleData</tt> your data folder while cropping.
+
To manually align the particles, do the following for about 10 particles:
 +
# Place the mouse over the particle center and press the key [c] (this centers the particle).
 +
# Place the mouse over its top (or bottom) part and press the key [n] (this aligns the particle).
 +
# Click on the particle to make sure its number turns from red to blue (blue means it is selected for further processing). To de-select a particle use the right-click.
 +
Change between the x-, y- and z-views to correct the orientations if necessary. This is just done to create an initial reference, so orientations don't need to be very exact.
  
* Load particles into memory by clicking “load” in the Load from disk field on the left of the window.  Note that at first, the scene will show only '''one''' particle.
+
[[File:StartersGuideGallery2.png|thumb|center|500px|]]
* Display some or all of the particles with the slider in the <tt>Shown</tt> particles field on the top of the window.
 
* Toggle between X- Y- and Z-views in the View: orientation field.
 
  
=== Manual alignment of particles in <tt>dgallery</tt> ===
+
Save the selected tags and corresponding table by clicking ''quick save'' button in the top right. It saves a quickbuffer.tbl and quickbuffer.tags to the hard drive that will be used later. To generate the average you need to apply the table on the particles. For this click the ''average'' button in the Particle selection field.
  
Now for each particle you can specify the center with a mouse click and the “C” button. Thermosomes are barrel-like particles with a long axis, you therefore may also specify the “top” of the particle by clicking along the long axis of the particle (but still within the box) with a mouse click and click “N” (stands for North). Do it for all the particles alternating the X- Y- and Z- views. Non-aligned particles are marked red, once the particle has been left-clicked it adds to the section in memory and turns blue. To remove from selection right click on the box.
+
[[File:StartersGuideGallery3.png|thumb|center|500px|]]
  
=== Selecting particles in <tt>dgallery</tt> ===
+
It opens a new window with a lot of controls. Click ''compute average'' in the bottom of the window. Then, right-lick on the output filename and click ''ok'' in the next window to see the result. If you are not satisfied with the result, close the window and refine your manual alignment and/or add more particles to the average.
  
You can save selected tags and a corresponding dynamo table by clicking the “quick save” button in the Particle Section field on top of the window. It saves a quickbuffer.tbl and quickbuffer.tags to the hard drive that you will use later.
+
[[File:StartersGuideGallery4.png|thumb|center|600px|]]
  
=== Averaging selected particles ===
+
[[File:StartersGuideGallery5.png|thumb|center|500px|]]
To generate the average you need to “apply” the table on the particles. For this click the “average” button in the Particle selection field. It opens dynamo_average_GUI with a lot of controls, we only need output filename in the Averaged volume field - my_average.em. Click “compute average” in the bottom of the window; wait till it is done. Right click to the output filename and select the [view] simple 3d depiction of all slices option and examine it from X- Y- and Z- views. If you are not satisfied with the result close the window and refine your manual alignment / add more particles to the average.
 
  
 
==Alignment projects==
 
==Alignment projects==
Iterative alignment of subtomograms to the average is performed by dynamo projects that you can run in various high-performance computational environments. To have a project you need particles, an initial reference and a table which you have already generated. Start a [[dcp GUI| WIZARD]] in the ''Dynamo'' command window or type
+
Iterative alignment of subtomograms to their average is performed by ''dynamo'' projects that you can run in various high-performance computational environments. To run an alignment project you need the particles, an initial reference and a table. All of these files we generated before. Start the alignment project GUI by typing
  
  <tt>>> dcp</tt>
+
  <tt> dcp</tt>
  
in Matlab (<tt>dcp</tt> stands for ''dynamo current project''). Type a project name “drun1” and press Enter, ''Dynamo'' will generate the auxiliary files in a folder called <tt>drun1</tt>.  
+
In thw new window, do the following:
 +
# Add the project name ''drun1'', press the ''enter'' key and select ''create a new project'' in the pop-up window.
 +
# Click on ''particles'' and provide the particle folder name ''thermosomeParticles.Data''. Click ok.
 +
# Click on ''table'' and provide the table name ''thermosomeParticles.Data/crop.tbl''. Click ok.
 +
# Click on ''template'' and provide the initial reference name ''my_average.em''. Click ok.
 +
# Click on ''masks'' and simply click on ''use default masks''. Click ok. You could also specify the semi-axis of the ellipsoid masks or go to “Mask editor” to make more sophisticated masks. Note that ''Dynamo'' by default uses Rossman correlation which eliminates the artefacts associated with hard (non-soft) mask.  
  
===Linked files===
+
[[File:StartersGuideAli1.png|thumb|center|500px|]]
  
Provide input in the popup windows: folder with particles (and press OK), initial table <tt>particles/crop.tbl</tt>. You can also provide the <tt>quickbuffer.tbl</tt> which has the initial orientations from manual picking, however remember that this table contains only the selected particles. In cases when you don’t have a table you can generate a blank or a random tables that would be consistent with your particle folder but will lack the metadata i.e. real missing wedge values or source tomograms. Go to “template” and input <tt>my_average.em</tt>.
+
Click on ''numerical parameters'' to set all parameters and details about the different iterations of the alignment project. You can select any parameter in the table with a mouse and click the ''?'' button on the top right of the parameter window to see a description of the parameters. The most important parameters to consider are:
Go to “Masks”, here you can provide several masks for different purposes, now we are only interested in the alignment mask. Use ellipsoid, you can specify semi-axis or go to “Mask editor” to make more sophisticated masks. Note that Dynamo by default uses Rossman correlation which eliminates the artefacts associated with hard (non-soft) mask. As usual you right click on the mask that you generate in order to open it with dview. For this type of simulated data “use default masks” will work.
 
  
===Numerical parameters===
+
* ''Number of iterations'': Make 3 rounds with 2 iterations each. The first round will be a global search with a coarse angular step and the following rounds will be used for refinement.
Numerical parameters menu provides flexibility on how to run your alignment. You can specify different parameters for different rounds of refinement. You can select any value in the table with a mouse and click the “?” button on the top right of the parameter window, this will give you more info. For more parameters check boxes on the left. The most important parameters to consider are:
+
* ''Angular search ranges'': Cone aperture is the scan range for the first two Euler angles around the initial orientation defined in the table. 360 degrees is the full scan range. Azimuth rotation range defines the rotation range around the new vertical axis of the particle.
 +
* ''High- and lowpass values'': Fourier voxels to limit the used frequency range.
 +
*''Particle dimensions'': Defined as sidelength of your subvolume. If you put a lower value, the particles will be downsampled for the particular round. This will speed up the process.
 +
* ''Refine'': After each angular scan, the search step is reduced by the ''refine factor''. This is repeated ''refine'' times. I.e., if your cone sampling is 10 degrees, ''refine'' is 3 and ''refine factor'' is 2, then 10, 5, 2.5 and 1.25 degrees will be sampled. This is the optimization of angular search space.
 +
*''Shift limits'': Limits the translation of particles from the center of the box (if shifts limiting way is 1) or from the previously estimated center (if shifts limiting way is 2). The previous estimates for the shifts and rotations are taken from input tables and are updated at the end of each iteration.
 +
*''Symmetry'': If you know the symmetry of your protein it will speed up the convergence and result in higher resolution.
  
* ''Number of iterations'' in each round. Make 2 rounds with 3-4 iterations in each. The first round will be global search with a course angular step, the second – constrained search with a finer step. If you decided to use a table with prealigned particles (quickbuffer.tbl) you may skip the first round.
+
Set the parameters as follows and click ''ok'':
* ''Angular search ranges'': cone aperture is the scan range for the first two Euler angles with a step of cone sampling starting at the actual angles stored in the table. 360 is the full scan range. Azimuth rotation range defines rotation range around the new vertical axis of the particle
 
* ''High- and lowpass values'' in Fourier voxels limit the used frequency range. It can be autotuned (below in this manual), here use some reasonable values like half-Nyquist.
 
*''Particle dimensions'' – use the sidelength of your box; if you put lower value the particles will be downsampled for the particular round. This will speed up the process.
 
* ''Refine'' after each angular scan the search step reduces refine factor times, this repeats refine times. I.e. if your cone sampling is 10 deg, refine and refine factor and 3 and 2 then 10, 5, 2.5 and 1.25 degrees will be sampled. This is the optimization of angular search space. We typically use 2 and 2.
 
*''Shift limits'' limits translation of particles from the center of the box (if shifts limiting way is 1) or from the previously estimated center (if shifts limiting way is 2). The previous estimates for the shifts and rotations are taken from input tables and are updated at the end of each iteration.
 
*''Symmetry'' – if you know the symmetry of your protein it will speed up the convergence and result in higher resolution. Here we don’t make any assumptions.
 
*''Threshold parameters and modus'' specify which particles contribute to the average at the end of each iteration. Our typical values are [1,2] and [0.5,5].
 
  
There are predefined parameter sets for Global and Local searches (''Predefined profiles -> Global search / Refinement''). After you set up the parameters press ''OK'' or close the window.
+
[[File:StartersGuideFigure10.png|thumb|center|600px|]]

 
[[File:StartersGuideFigure10.png|thumb|center|600px| Figure 10. Summary of numerical parameters used for sub tomogram averaging.]]
 
  
Go to Computing environment in Wizard and select Matlab-CPU. Click “Info (nvidia)” it should output you a table to your Matlab window, your GPU identifiers are in the left column of the table. Put those into the GPU identifiers to the ‘identifiers” field.
+
Click on ''Computing environment'' and select your computing environment (choose ''standalone'' if you are working on the standalone version during a workshop). Set ''CPU cores'' and ''parallelize averaging step'' to the maximum available (e.g., 24). Leave the rest as it is and click ''ok''. In the alignment GUI click ''check'' and ''unfold''. To run the project
  
===Running the project===
+
* in a Matlab session, you can just click on “run”. 
 +
* in a standalone project it is more convenient to open a new terminal, load ''Dynamo'' (but do not run it) and run the project by typing <tt>./drun1.exe</tt>.
  
Go back to the Wizard window, click “check” and ‘unfold”, this will generate a runnable matlab script drun1.m. If you will modify your project and will want to run it again you need to re-unfold the project before each run.
+
While the project is running, open another Matlab window (or in standalone open another terminal with ''Dynamo'' running) and you can monitor the progress of the alignment project by typing:
  
* In a Matlab session you can just click on “run”.  
+
  <tt>dvstatus drun1</tt>
* In a standalone project it's more convenient to open a new terminal, activate ''Dynamo'' on it and run the project just invoking the name of the project script that was generated by the unfolding step:
 
  
<tt> ./drun1.exe</tt>
+
While the project is running, you can look at projections of the intermediate results by typing:
  
Open another Matlab window, activate dynamo, go to the same folder and monitor the progress of the execution by typing in >> dvstatus drun1. After several iterations you can look at intermediate results by typing:
+
<tt>ddb drun1:a:ite=* -j c10</tt>
  
  <tt> >> ddb drun1:a –v </tt>
+
or at the latest average by typing:
 +
 
 +
  <tt>ddb drun1:a -v </tt>
 
   
 
   
or
+
The final result should look similar to:
 
 
<tt>>>ddb drun1:a:ite=* -j c10</tt>
 
 
 
For more info on <tt>ddb</tt> type <tt> >> help ddb</tt>. In the end of your iterations you can monitor the results in the show results section of Wizard and should get something like this:
 
  
 
[[File:StartersGuideResultsAlignment.png|thumb|center|400px| Results of alignment.]]
 
[[File:StartersGuideResultsAlignment.png|thumb|center|400px| Results of alignment.]]
Line 143: Line 150:
  
 
=Subtomogram classification=
 
=Subtomogram classification=
First generate a dataset which has 2 populations of particles. For this, type:
+
To demonstrate a classification example we provide particles with 2 slightly different sizes. We can then classify them with a simple Multi-reference Analysis (MRA). To do that, we first generate a dataset which has 2 populations of particles. For this, type:
  
  <tt>>> dynamo_tutorial('data_classification','M',8,'N',8);</tt>
+
  <tt>dynamo_tutorial('data_classification','M',8,'N',8);</tt>
  
in your Matlab window. It generates two sets of particles, 8 particles in each and stores their real orientations in the directory <tt>data_classification</tt>.
+
It generates two sets of particles, 8 particles in each including their table. The idea behind [[Multireference Analysis|MRA]] is the following: The particles are aligned to several references (here only 2). After each iteration, each particle is assigned to only one single reference where it fits the best. This procedure is repeated iteratively. Similar particles will have higher correlation to similar references and will eventually group together. For classification into N classes we will need N initial references and N initially identical tables.Follow these steps to run the MRA:
  
==Multireference alignment==
+
# Create a new project by typing “drun2” in the project name of the alignment GUI.
The idea behind [[Multireference Analysis|MRA]] is: iterative alignment of particles to several references and then each particle assigned to the reference where it fits the best. This way over iterative refinement similar particles will have higher correlation to similar references and will eventually group together. MRA is naturally implemented in ''Dynamo''. For classification into N classes we will need N initial references and N initially identical tables that ''Dynamo'' will generate automatically. The references should be slightly different to start particle differentiation, in ''Dynamo'' we add a small amount of white noise to the overall average of all particles.
+
# Set number of references to 2.
 
+
# Activate the ''Swap particles'' button.
=== Creating MRA projects ===
+
# Go to ''particles'' and add <tt>data_classification/data</tt> to the particle data. Click ok.
Create a new project by typing “drun2” in the project name of Wizard set number of references to 2, set ''Swap particles'' on, put in the particle data.  
+
# Go to ''table'' and type <tt>data_classification/real.tbl</tt> into the field ''clone this'' and press ''copy''. Click ok.
+
# Go to ''template'' and clone <tt>data_classification/original_template.em</tt> with addition of some extra noise with amplitude 1.  
=== Seeds for MRA projects ===
+
# Click on ''masks'' and click on ''use default masks''. Click ok.
* Go to “table” and type <tt>data_classification/real.tbl</tt> into the “clone this” field and press “copy”, press “OK”.
+
# Click on ''numerical parameters'' and set the following parameters. Click ok.
* Go to “template” and clone data_classification /original_template.em with addition of some extra noise with amplitude 1.  
 
* Initialize masks with the defaults parameters (large button on the bottom).  
 
  
Go to “numerical parameters” and use Predefined profiles -> refinement. Increase angular search steps to 5, set a thresholing policy and press “OK”. Set computing environment to GPU (under Matlab), specify GPU identifiers.  
+
[[File:StartersGuideResultsClassification2.png|thumb|center|400px|]]
  
=== Execution of projects ===
+
Choose the same computing environment as in the previous alignment project and run the project in the same way too. MRA projects usually run longer, because each particle will be aligned to two references. Here, however, we chose the parameters in a way that the process is still fast (e.g., small dimensions). Monitor the progress of the project same as before:
Check, unfold and run the project, this project will run twice longer as each particle will be aligned to two references. Monitor the progress of your project by
 
  
  <tt>>> dvstatus drun2</tt>
+
  <tt>dvstatus drun2</tt>
  
and visualize the results as
+
And visualize the results using the command:
  
  <tt> >> ddb drun2:a:ref=* -j c5</tt>
+
  <tt>ddb drun2:a:ref=* -j c5</tt>
  
You should get something like this – two averages with particle in one class larger than in the other.
+
You should get two averages with particle in one slightly class larger than in the other.
  
 
[[File:StartersGuideResultsClassification.png|thumb|center|600px| Results of classification.]]
 
[[File:StartersGuideResultsClassification.png|thumb|center|600px| Results of classification.]]

Latest revision as of 18:27, 3 September 2021

This tutorial shows how to manually pick particles in a tomogram, extract them, align and classify them. This tutorial is a shortened version of the original starters guide that was adapted for the use in Dynamo workshops.

Manual particle picking and Dynamo catalogues

The Dynamo catalogues are databases that manage tomograms and link the tomographic data to the extracted particles. Start the catalogue manager by typing the command:

dcm

After the catalogue manager opens, we create 3 synthetic tomograms that include our particles (thermosomes) in the following way:

Figure 1: creating a set of synthetic tomograms with Dynamo Catalogue Manager

You can see the list of your tomograms and their metadata in a table in the bottom of your catalogue manager. When workin on your own projects, you can add tomograms to the catalogue by Catalogue -> Browse for new volume. The tomograms are situated in the directory testCatalogue. They contain a small amount of noise and a missing wedge associated to rotation around Y-axis. We added the information about location of the particles in the tomograms in an additional catalogue called testCatalogue_withmodels. If needed, it can be opened from the current folder with Catalogue -> look for local catalogues in the catalogue manager window.

Viewing tomograms

Select the first tomogram in the list and go to View volume>Full tomogram file in tomoslice.

Figure 2: Selecting a volume in the catalogue manager to be inspected in dpreview

The volume browser tomoslicer loads the entire tomograms into memory and allows making annotations to the regions of interest. The tomograms in this tutorial are small, and you can load them directly into memory. For real life tomograms, bear in mind that you will need to prebin the files before loading them into memory. Tomoslice has a simple set of controls and is suitable for visualization tasks that require oblique sections through the tomogram. It uses the same tool as other Dynamo browsers to keep track of your annotations: a pool of models. You might need to adjust the contrast (blue arrow). To move through the tomogram slices, you can either use the mouse wheel, click and drag the tomogram slice up and down (orange arrows), or move the position control left and right (orange box).


StartersTomoSlice.png

Picking and extracting particles in tomograms

Coordinates of picked particles are represented by data types called models. In the tomoslicer window go to Model Pool -> Create new model in pool (choose type) -> General. This is the simplest type of model where each clicked/model point corresponds to a single isolated particle. Now you can navigate up and down the tomogram, place the mouse on the center of a particle and press the [c] key on your keyboard to add a new model point. (Note the Help -> all hot keys options that lists the different the actions of the different keystrokes). Backspace button deletes the last clicked point (you can also use the right-click to delete single points).

Generating a new model

After you are done clicking about 10 particles click on Active model > Update Crop points in the tomoslice window. 

StartersGuideUpdate.png

Save the model into the catalogue by Active model -> Save active model into catalogue (disk) and close the slicer window.

StartersGuideFigure6.png

Pick particles for tomograms 2 and 3. In total, there are around 30-40 particles in the generated dataset. When you open a new tomogram, make sure that you delete the pool of models from memory when asked. This will have no effect onto the models stored in disk, and it is necessary in order to ensure that you are not mixing models from different tomograms.

An alternate visualization: orthogonal projections

Clicking in the menu on Projection -> project full shown fragment along z (still in dtmslice) and you will get a screen where the x-y projection of the tomogram is shown. Use the secondary click on it to launch the orthogonal views of x-z and y-z planes that traverse that point. These views can also be used to click particles, in case the standard view is not sufficient.

click on the main projection determines two orthogonal slices
Slice xz
Slice xy

Extracting particles from tomograms

In the catalogue manager window select the rows for tomograms from which you want to extract particles from. You can either select them one by one with a mouse click and holding the [ctrl] key or by clicking the Select all button. Then, go to Crop Particles -> Open Volume List Manager.

StartersGuideopenVolumeList.png

A new window opens with all the models in the catalogue listed in the bottom. Pick (by checking boxes) the models of type general that you have just clicked. Click Create list and then Crop particle.

StartersGuideSelectModels.png

In the new window, change the data name to something meaningful, such as thermosomeParticles, change the sidelength from 32 to 48 and click start cropping.

StartersGuideCrop.png

After cropping is done, explore your cropped particles by clicking ddbrowse (in the window above). A new window opens where you can simply click on show.

StartersGuideShow.png

You see a 2D representation (projections) of all cropped 3D subtomograms. Make sure your particles are well centered and fit the box.

StartersGuideParticles.png

We also introduce the concept of the dynamo table: The information about each particle is stored in tables. Each particle has an entry in the table that contains shifts and rotations to describe its orientation (by default these values are initialized as zeroes). It also contains the particle ID (tag), the orientation of the missing wedge and others. For full info type the command dthelp. The Dynamo catalogue generates an initial table during particle extraction. Its location is in particles/crop.tbl .

Subtomogram alignment and averaging

Initial reference generation

We want to align the extracted tomograms to a common reference. For that, we need an initial reference (which will later be refined during the iterative alignment). Here, we create such an initial reference by manually align a few particles and average them. We use the command dgallery to generate initial orientations for some of our particles that will be used to generate the initial reference. Close all open Dynamo windows and type into the Dynamo command line:

dgallery('data','thermosomeParticles.Data');

The gallery opens. Click on load to load all aprticles in memory, move the shown bar to display the particles and use the x-, y- and z-buttons to see different views:

StartersGuideGallery.png

To manually align the particles, do the following for about 10 particles:

  1. Place the mouse over the particle center and press the key [c] (this centers the particle).
  2. Place the mouse over its top (or bottom) part and press the key [n] (this aligns the particle).
  3. Click on the particle to make sure its number turns from red to blue (blue means it is selected for further processing). To de-select a particle use the right-click.

Change between the x-, y- and z-views to correct the orientations if necessary. This is just done to create an initial reference, so orientations don't need to be very exact.

StartersGuideGallery2.png

Save the selected tags and corresponding table by clicking quick save button in the top right. It saves a quickbuffer.tbl and quickbuffer.tags to the hard drive that will be used later. To generate the average you need to apply the table on the particles. For this click the average button in the Particle selection field.

StartersGuideGallery3.png

It opens a new window with a lot of controls. Click compute average in the bottom of the window. Then, right-lick on the output filename and click ok in the next window to see the result. If you are not satisfied with the result, close the window and refine your manual alignment and/or add more particles to the average.

StartersGuideGallery4.png
StartersGuideGallery5.png

Alignment projects

Iterative alignment of subtomograms to their average is performed by dynamo projects that you can run in various high-performance computational environments. To run an alignment project you need the particles, an initial reference and a table. All of these files we generated before. Start the alignment project GUI by typing

 dcp

In thw new window, do the following:

  1. Add the project name drun1, press the enter key and select create a new project in the pop-up window.
  2. Click on particles and provide the particle folder name thermosomeParticles.Data. Click ok.
  3. Click on table and provide the table name thermosomeParticles.Data/crop.tbl. Click ok.
  4. Click on template and provide the initial reference name my_average.em. Click ok.
  5. Click on masks and simply click on use default masks. Click ok. You could also specify the semi-axis of the ellipsoid masks or go to “Mask editor” to make more sophisticated masks. Note that Dynamo by default uses Rossman correlation which eliminates the artefacts associated with hard (non-soft) mask.
StartersGuideAli1.png

Click on numerical parameters to set all parameters and details about the different iterations of the alignment project. You can select any parameter in the table with a mouse and click the ? button on the top right of the parameter window to see a description of the parameters. The most important parameters to consider are:

  • Number of iterations: Make 3 rounds with 2 iterations each. The first round will be a global search with a coarse angular step and the following rounds will be used for refinement.
  • Angular search ranges: Cone aperture is the scan range for the first two Euler angles around the initial orientation defined in the table. 360 degrees is the full scan range. Azimuth rotation range defines the rotation range around the new vertical axis of the particle.
  • High- and lowpass values: Fourier voxels to limit the used frequency range.
  • Particle dimensions: Defined as sidelength of your subvolume. If you put a lower value, the particles will be downsampled for the particular round. This will speed up the process.
  • Refine: After each angular scan, the search step is reduced by the refine factor. This is repeated refine times. I.e., if your cone sampling is 10 degrees, refine is 3 and refine factor is 2, then 10, 5, 2.5 and 1.25 degrees will be sampled. This is the optimization of angular search space.
  • Shift limits: Limits the translation of particles from the center of the box (if shifts limiting way is 1) or from the previously estimated center (if shifts limiting way is 2). The previous estimates for the shifts and rotations are taken from input tables and are updated at the end of each iteration.
  • Symmetry: If you know the symmetry of your protein it will speed up the convergence and result in higher resolution.

Set the parameters as follows and click ok:

StartersGuideFigure10.png

Click on Computing environment and select your computing environment (choose standalone if you are working on the standalone version during a workshop). Set CPU cores and parallelize averaging step to the maximum available (e.g., 24). Leave the rest as it is and click ok. In the alignment GUI click check and unfold. To run the project

  • in a Matlab session, you can just click on “run”.
  • in a standalone project it is more convenient to open a new terminal, load Dynamo (but do not run it) and run the project by typing ./drun1.exe.

While the project is running, open another Matlab window (or in standalone open another terminal with Dynamo running) and you can monitor the progress of the alignment project by typing:

dvstatus drun1

While the project is running, you can look at projections of the intermediate results by typing:

ddb drun1:a:ite=* -j c10

or at the latest average by typing:

ddb drun1:a -v 

The final result should look similar to:

Results of alignment.

Subtomogram classification

To demonstrate a classification example we provide particles with 2 slightly different sizes. We can then classify them with a simple Multi-reference Analysis (MRA). To do that, we first generate a dataset which has 2 populations of particles. For this, type:

dynamo_tutorial('data_classification','M',8,'N',8);

It generates two sets of particles, 8 particles in each including their table. The idea behind MRA is the following: The particles are aligned to several references (here only 2). After each iteration, each particle is assigned to only one single reference where it fits the best. This procedure is repeated iteratively. Similar particles will have higher correlation to similar references and will eventually group together. For classification into N classes we will need N initial references and N initially identical tables.Follow these steps to run the MRA:

  1. Create a new project by typing “drun2” in the project name of the alignment GUI.
  2. Set number of references to 2.
  3. Activate the Swap particles button.
  4. Go to particles and add data_classification/data to the particle data. Click ok.
  5. Go to table and type data_classification/real.tbl into the field clone this and press copy. Click ok.
  6. Go to template and clone data_classification/original_template.em with addition of some extra noise with amplitude 1.
  7. Click on masks and click on use default masks. Click ok.
  8. Click on numerical parameters and set the following parameters. Click ok.
StartersGuideResultsClassification2.png

Choose the same computing environment as in the previous alignment project and run the project in the same way too. MRA projects usually run longer, because each particle will be aligned to two references. Here, however, we chose the parameters in a way that the process is still fast (e.g., small dimensions). Monitor the progress of the project same as before:

dvstatus drun2

And visualize the results using the command:

ddb drun2:a:ref=* -j c5

You should get two averages with particle in one slightly class larger than in the other.

Results of classification.