Difference between revisions of "Getting started"

From Dynamo
Jump to navigation Jump to search
(Created page with " There are two volume browserstomoviewand tomoslicerthat load the entire tomograms into memory and allow making annotations to theregions of interest. Tomoview...")
 
 
(14 intermediate revisions by the same user not shown)
Line 1: Line 1:
 +
The best way to learn ''Dynamo''on your own is to follow the materials in one of our 2 day-workshops [[EMBO workshop 2020: this one]]
  
There  are  two  volume  browserstomoviewand tomoslicerthat load  the entire  tomograms  into memory  and allow  making  annotations  to theregions  of interest.  Tomoview  provides  fine-tuned visualization  along  orthogonal  directions. Tomoslice has  simpler  set  of  controls, and is also more suitable  for  visualization  tasks  that  require  oblique  sections  through  the  tomogram.    While thevisualization is different, both browsersuse the same toolto keep track of your annotations: a poolof models.Let’s exploreit with the simpler tomoslice.Go  to View  volume ->  load  full  tomogram  into Tomoslice.On  the  left  control  panels Sceneturn  off the 3d view radiobutton. You can navigate the z-height of the tomogram with a mouse wheel or with a  slider  in  the tomoslicemenu  (1  to  256  in  this  case);  you can also  change  the  slice  thickness  from the default 10pixels. Play with the controls and explore the tomogram.Turn 3d view back on.PICKING PARTICLES IN TOMOGRAMSCoordinates  of  picked  particles  are  represented  by data  types  called models.  In  the tomoslicer window go to Model Pool -> Create new model in pool (choose type) -> General. This is the simplest type of model where each clicked/model point corresponds to a single isolated particle. Now you can navigate  up  and  down the tomogram, center the mouse on the particle and click “C” to add a new model  point. (Note  the  Help ->  Shortcut  options  that  lists  the  different  the  actions  of  the  different keystrokes). Backspace button deletes the last clicked point. Figure 4. Generating a new modelAfter  you  are  done  clicking  a  dozen  of  particles  go  to Active  model ->  Edit  active model. This  is  a summary of the model with control elements for geometric transformations. Models of type generaldon’t allow any geometrical transforms, all you need to do here  is  to  convert  the  clicked  model points into “table points” that will be used for cropping. This is done by clicking the table-looking command  icon  (4thfrom  the  right  in the  top  left  part  of  the Editor  of  model  parameterswindow).  Alternatively, you could just click on Active model > Update Crop points in the tomoslicewindow.
+
 
Figure 5. Updating crop points and extracting particles from the model editor windowThe model editor also gives you the possibility of changing the name of the model. You may want to give themodels  meaningful identifiersbefore  storing  them  into  disk.Also,  you  can  open  a  GUI  for extracting  your  particles  (first  icon  from the  right)  to  extract  particles  from  only  this  tomogram, however we will do it later for all 3 tomograms. Make sure that the number of clicked points equals to the number of table points in the Informationfield and close the Editorwindow. Importantly, save the model into the catalogue by Active model -> Save active model into catalogue (disk)and close the slicerwindow.  Pick  particles  for  tomograms  2  and  3. In  total,  there  are around  40particles  in  the generated dataset.Figure 6.Remember to save your work: use Update Crop to create table points, and then save the model into the catalogue
+
== Starters guide ==
Important:  when  you opena  new  tomogram,  make  sure  that  you  delete  the  pool  of  modelsfrom memory  (model  pool  >  clear  current  pool  from  memory). This will  have  no  effect  onto  the  models stored  in  disk,  and  it  is necessary  in  order  to  ensure that  you  are  not  mixing  models  from  different tomograms. AN ALTERNATIVE VISUALIZATION:TOMOVIEWThe just described tomoslicerbrowser is one  of a visualization option  for  tomograms.  You  canuse alternatively the tomoviewbrowser(or skip this section if you are already satisfied with tomoslice). Tomoviewcan be activated through the preview window in the same wayas we activated tomoslice.Tomosliceforces you to move the viewed slice up and down in a tomogram in order to reliably locate its center. In tomoviewyou will click two times on each particle: once on the projection through the tomogram  in  Z-direction to  locate  the  (x,y)  center.  Second –in  the  orthogonal  to  the  projection direction tolocate the Z-height.Click on Picking profile > Isolated particles.This will create a model of type “General” (no  need  to  create  it  specifically  with  the  [Model  Pool] window),  and also will  tune the visualization options so that the main window shows a projection of the whole tomogram along z.In the field external viewsactivate the Localradiobutton and type in 200; in the Annotationsfield make sure supportis on. Right (secondary) click on the particle in the tomogram (this is a projection in the Z direction) to pick X-and Y-coordinates. To center the Z-position of your particles left click in the y-slice window. Help on clicking is available under Help.Figure 7tomoview  tuned  forpickingisolated  particles.  Right: external  viewer:  a  part  of  the  yz plane including the [x,y] position clicked in the main windowEXTRACTING PARTICLES FROM TOMOGRAMSIn  the catalogue  managerwindow  select  the  rows  for  tomograms from  which  you  want  to  extract particlesfrom.  You  can  either  select  them  one  by  one  with  a  mouse  click  holding  a  CTRL  button  or click the “Select all” button.
+
Concerning the task of subtomogram averaging, you can get a plunge-freeze immersion in ''Dynamo'' for from scratch, by following the [[Starters guide| starters guide]]. This is a short hands-on complete walkthrough through the main functionalities in ''Dynamo'': [[Viewing tomograms|viewing]] and [[Catalogue|organizing]] tomograms, [[Particle extraction|extracting particles]], [[Subtomogram alignment|aligning]], averaging and [[Classification|classifying]] subtomograms, and [[Adaptive bandpass filtering|avoiding overfitting]].
Figure 8.Extracting particles with DCM and selectingthe models for particle extractionGo  to Crop  Particles ->  Open  Volume  List  Manager, all  the  models  in  the  catalogue  listed  in  the bottom of the window. Pick (by checking boxes) the models of type general –the ones that you have just  clicked.  Increase  sidelength  to  48-64  pixels.  Vll-list  is  the  format  in  which  Dynamo  stores directives  for  particle  extraction;  you  can  modify  it  with  your  own  scripts  if  you  have  a  lot  of
+
 
tomograms. Now click “Create list” and “Crop particles”. Change output directory name to i.e. “particles” and click “start cropping”.Figure 9.Cropping particles from the set of selected modelsAfter cropping is done explore your cropped particles by clicking ddbrowseunder the output  folder name. Ddbrowse  (Dynamo Data Browse)  is a lightweight  browser for particle visualization, just click “show” and make sure your particles are well centeredandfit  the  box.  If  not -use  pre-existing models to  generate the  vll-file  and  re-extract  the  particles  or  increase  the  sidelength.Please  note that Dynamo generated a particles/folder with sub-volumes formatted as particle_000XX.em.
+
 
SUB-TOMOGRAM ALIGNMENT AND AVERAGINGINITIAL MODEL GENERATIONIf you don’t have an initial model it is easy to generateitby manually aligning several particles  and summing  them  up.  Dynamo  stores  information  about  the  particles  in  dynamo tables:  for  each particle a table  contains  shifts  and  rotations  to  bring  the  particle  to  the  average  (by  default these values are initialized as 0s), particle ID (tag), orientation of the missing wedge and others. For full info type >> dthelpin Matlab. Dynamo catalogue generates aninitial table during particle extraction, its location  is  particles/crop.tbl.  For  initial  model generation we  will  use dynamo_galleryto  generate initial orientations for some of our particles. Type >> dgallery(‘data’, ‘particles’, ‘table’, ‘particles/crop.tbl’);Load particles into memory by clicking “load” in the Load from diskfield on the left of the window. Display some or all of the particles with a bottom slider in the Shown particlesfield on the top of the window.Toggle between X-Y-and Z-viewsintheView: orientationfield. Now for each particle you can specify the center with a mouse click and the “C” button. Thermosomes are barrel-like particles with a long axis, you therefore may also specify the “top” of the particle by clicking along the long axis of the particle (but still within the box) with a mouse click and click “N” (stands for North). Do it for all the particles alternating the X-Y-and Z-views.Non-aligned particles are marked red, oncethe particle  has  been left-clicked it  adds to  the  section  in  memory  and  turns  blue.  To  remove  from selection  right  click  on  the  box.  You  can  save  selected  tags  and  a  corresponding  dynamo  table  by clicking the “quick save” button in the Particle  Sectionfield  on  top  of  the window.  It  saves  a quickbuffer.tbland quickbuffer.tagsto the hard drive that you willuse later. To generate the average you need to “apply”the table on the particles.For this click the “average” button  in  the Particle  selectionfield.  It opens dynamo_average_GUIwith  a  lot  ofcontrols, we only need  output  filename  in  the Averaged  volumefield  -my_average.em. Click “compute average” in the bottom of the window; wait till it is done. Right click to the output filename and select the [view] simple 3d depiction of all slicesoption and examine it from X-Y-and Z-views. If you are not satisfied with  the  result  close  the  window  and  refine  your  manual  alignment/  add  more  particles  to  the average.ALIGNMENT PROJECTIterative alignment of sub-tomograms to the average is performed by dynamo projectsthat you can run in various high-performance computational environments. To have a project you need particles, an  initial  reference  and a  table  which  you  have  already  generated.  Start  a  WIZARD  in  the  Dynamo command window or type >> dcpin Matlab (dcp stands for dynamo_current_project). Type a project name “drun1”and  press  Enter,  dynamo  will  generate  the  auxiliary  files  in  a  folder  drun1.  Provide input in the popup windows: folder with particles (and press OK), initial table particles/crop.tbl. You can also provide the quickbuffer.tblwhich has the initial orientations from manual picking, however remember that this table contains only the selected particles. In cases when you don’t have a table you can generate  a blank or a random tables that would be  consistent with your particle folder but will  lack  the  metadata  i.e.  real  missing  wedge  values  or  source  tomograms.Go to “template” and input my_average.em.
+
== Getting help ==
Go to “Masks”, here you can  provide  several  masks  for  different  purposes,  now  we  are  only interested in the alignment mask. Use  ellipsoid, you can specify  semi-axis or go to “Mask editor” to make  more  sophisticated  masks.  Note  that  Dynamo  by  default  uses  Rossman  correlation  which eliminates  the  artefacts  associated  with  hard  (non-soft)  mask.  As  usual  you  right  click  on  the  mask that you generate in order to open it with dview. For this type of simulated data “use default masks” will work.Numerical  parameters menu  providesflexibilityon  how  to  run  your  alignment.You  can  specify different parameters for different rounds of refinement. You can select any value in the table with a mouse and click the “?” button on the top right of the parameter window, this will give you more info. For more parameters check boxes on the left. The most important parameters to consider are: Number  of  iterations  in  each  round.  Make  2  rounds  with  3-4  iterations  in  each.  The  first round will be global searchwith a course angular step, the second –constrained search with a finer step.If you decided to use a table with prealigned particles (quickbuffer.tbl) you may skip the first round.Angular  search  ranges: cone  apertureis the  scan  range  for  the  first  two  Euler angles  with  a step  of cone  samplingstarting  at  the  actual  angles  stored  in  the  table.  360  is  the  full  scan range. Azimuth  rotation  rangedefines  rotation  range  around  the  new  vertical  axis  of  the particle. High-and  lowpass  values  in  Fourier  voxels  limit  the  used  frequency  range.  It  can  be autotuned (below in this manual), here use some reasonable values like half-Nyquist.Particle dimensions –use the sidelength of your box; if you put lower value the particles will be downsampled for the particular round. This will speed up the process.Refineafter each angular scan the search step reduces refine factortimes, this repeats refinetimes. I.e. if your cone sampling is 10 deg, refineand refine factorand 3 and 2 then 10, 5, 2.5 and 1.25 degrees will be sampled.This is the optimization of angular search space. I typically use 2 and 2.Shift limits limits translation of particles from the center of the box (if shifts limiting wayis 1) or  from  the  previously  estimated  center  (if shifts  limiting  wayis  2).  The  previous  estimates for the shifts and rotations are taken from input  tables  and are updated at  the  end of each iteration.Symmetry –if you know the symmetry of your protein it will speed up the convergence and result in higher resolution. Here we don’t make any assumptions.Threshold  parameters and modus specify  which  particles  contribute  to  the  average  at  the end of each iteration. Ourtypical values are [1,2] and [0.5,5].There  are  predefined  parameter  sets  for  Global  and  Local  searches  (Predefined  profiles ->  Global search / Refinement).Afteryou set up the parameters press OK or close the window.
+
''Dynamo'' offers several ways of [[Getting help|getting help]] on specific subjects.... and of course, you can always [[https://groups.google.com/forum/#!forum/dynamo-for-cryo-electron-tomography contact our forum in Google Groups]]
Figure 10. Summary of numerical parameters used for sub tomogram averaging.Go to Computing environmentin Wizard and select GPU (with Matlab). Click “Info (nvidia)” it should output you a table to your Matlabwindow, your GPU identifiers are  in the left column of the table. Put those into the GPU identifiers to the ‘identifiers” field. Go back to the Wizardwindow,  click “check” and ‘unfold”, this will generate a runnable matlab script drun1.m. If  you  will  modify  your project and will want to run it again you need to re-unfold the project before each run.Click “run”. Open another Matlab window, activate dynamo, go to the same folder and monitor the progress of the  execution  by  typing  in >>  dvstatus  drun1.  After  several  iterations you  can  look  at  intermediate results by typing >> ddb drun1:a –vor >>ddb drun1:a:ite=* -j c10. For more info on ddb type >> help ddb. In the end of your iterations you can monitor the results in the show resultssection of Wizardandshould get something like this.
 
SUB-TOMOGRAM CLASSIFICATIONFirst generate    a    dataset    which    has    2    populations    of    particles.    For    this    type >> dynamo_tutorial('data_classification','M',8,'N',8); in  your  matlabwindow.  It  generates  two  sets  of particles, 8 particles in each and stores their real orientations in the directory data_classification.MULTI REFERENCE ALIGNMENT (MRA)The idea behind MRA is:iterative alignment of particles to several references and then each particle assigned to the  reference  where  it  fits  the  best. This  wayover  iterative  refinementsimilar  particles will have higher correlation to similar references and will eventually group together. MRA is naturally implemented  in  Dynamo. For  classification  into  N  classes  we  will  need  N  initial  references  and  N initially  identical  tables  that Dynamo  will  generate  automatically.  The  references  should  be  slightly different  to  start  particle  deconvergence,  in  Dynamo  we  add  small  amount  of white noise  tothe overall average of all particles.Create a new project by typing “drun2” in the project name of Wizardsetnumber of references to 2, set Swap particles on, put in the particle data. Go to “table” and type data_classification/real.tblinto the “clone this” field and press “copy”, press “OK”. Go to “template” and clone data_classification/original_template.emwith additionof some  extra noise with amplitude 1. Initialize  masks with the defaultsparameters (large button on the bottom). Go to “numerical parameters” and use Predefined profiles -> refinement. Increase angular search steps to 5, seta thresholing policyand press “OK”. Set computing environment to GPU (under Matlab), specify GPU identifiers.Unfold and run the project, this  project  will  run  twice longeras  each  particle  will  be  aligned  to  two  references.Monitor  the progress of your project by >> dvstatus drun2and visualize the results as >> ddb drun2:a:ref=* -j c5. You should get something like this –two averages withparticle in one class larger than in the other.CLASSIFICATION WITH PRINCIPAL COMPONENT ANALYSIS AND K-MEANSAn analytical method is described in a tutorial classification_tutorial.pdf, please refer to it with dhelp.INDEPENDENT HALF-DATASET REFINEMENTAligning    all    particles    to    one    reference    may    introduce    overfitting    resulting    in    resolution overestimation and a risk of overinterpretation of the final map. This is illustrated by a small exercise in  the noise_artifacts.pdftutorial. For  the  generation  of  the  final  map  it  is  therefore highly recommended  to  separate  your  particles  into  two  halves  and  perform  independent  alignment  and
 
averaging.  At  the  end  of  each  iteration  you can detect the current  resolution  by Fourier  shell correlation and only use your reliable frequencies for the alignment at the next steps. To  perform  independent  half-dataset  refinement  load  your  initial  project drun1to  Wizard  (type  in the Project field and press Enter or use Ctrl+L). Go to Multireference -> Adaptive filtering -> Derive a project,  Dynamo  will  generate  a  new  project drun1_eo, load  it,  inspect/update  the  parameters  and run  it. After  it  is  done  you  can  measure  your  resolution  based  on  fsc  plots  from Multireference -> Adaptive filtering -> Plotattained resolutionand inspect the average the usual way.
 

Latest revision as of 10:08, 26 February 2021

The best way to learn Dynamoon your own is to follow the materials in one of our 2 day-workshops EMBO workshop 2020: this one


Starters guide

Concerning the task of subtomogram averaging, you can get a plunge-freeze immersion in Dynamo for from scratch, by following the starters guide. This is a short hands-on complete walkthrough through the main functionalities in Dynamo: viewing and organizing tomograms, extracting particles, aligning, averaging and classifying subtomograms, and avoiding overfitting.


Getting help

Dynamo offers several ways of getting help on specific subjects.... and of course, you can always [contact our forum in Google Groups]