Difference between revisions of "Integration with Warp and M"
AlisterBurt (talk | contribs) |
AlisterBurt (talk | contribs) |
||
| (14 intermediate revisions by the same user not shown) | |||
| Line 5: | Line 5: | ||
* robust CTF estimation procedures for tilt-series data | * robust CTF estimation procedures for tilt-series data | ||
Dynamo's | Dynamo's | ||
| − | * automated tilt-series alignment | + | * automated fiducial based tilt-series alignment |
* geometric picking tools | * geometric picking tools | ||
* flexible subtomogram averaging procedures | * flexible subtomogram averaging procedures | ||
| Line 22: | Line 22: | ||
* Run alignments/classifications in dynamo or relion | * Run alignments/classifications in dynamo or relion | ||
* Run multi-particle refinements in M | * Run multi-particle refinements in M | ||
| + | * (optional) Rerun alignments/classifications in dynamo or relion | ||
The last steps can be repeated to further improve final reconstruction quality | The last steps can be repeated to further improve final reconstruction quality | ||
| Line 36: | Line 37: | ||
= align tilt-series = | = align tilt-series = | ||
| − | + | '''Without Fiducials''' | |
| − | A function to automate the alignment of datasets with fiducials and convert all necessary metadata for import into warp is provided as | + | Tilt-series without fiducials should be aligned in IMOD with [https://bio3d.colorado.edu/imod/doc/patchTrackExample.html patch-tracking ] |
| + | |||
| + | '''With Fiducials''' | ||
| + | |||
| + | Tilt-series can either be aligned in IMOD or using dynamo's new [[Walkthrough on command line based tilt series alignment | tilt series alignment tools]] | ||
| + | |||
| + | A function to automate the alignment of datasets with fiducials and convert all necessary metadata for import into warp is provided as [https://github.com/alisterburt/autoalign_dynamo autoalign_dynamo]. | ||
Once you've followed the setup from that page, simply navigate to the imod directory generated by warp and run | Once you've followed the setup from that page, simply navigate to the imod directory generated by warp and run | ||
| Line 53: | Line 60: | ||
Choose to also generate a deconvolved version, we will use this for visualisation. | Choose to also generate a deconvolved version, we will use this for visualisation. | ||
| + | |||
| + | Generate a [[Catalogue | catalogue]] using the function ''warp2catalogue()''provided in [[https://github.com/alisterburt/autoalign_dynamo autoalign_dynamo]] | ||
| + | <nowiki>warp2catalogue(<warp_reconstruction_folder>, <pixel_size_angstrom>)</nowiki> | ||
= pick particles in dynamo = | = pick particles in dynamo = | ||
| + | Pick particles using your favourite [[Model | type of model]] through the catalogue created in the previous step. | ||
| + | |||
| + | Make sure to crop particles from the catalogue directly, this will make sure that even though you picked particles on filtered volumes for visualisation the particles are cropped from the unfiltered tomograms. | ||
| + | |||
| + | You can choose to run first sets of alignments on these downscaled particles to find which particles you would like to use for later refinements. | ||
= reextract unbinned particles in warp = | = reextract unbinned particles in warp = | ||
| + | Once you have a set of positions and orientations you want to use in refinements at a smaller pixel size, we need to convert this metadata into something Warp can understand so it can reconstruct our particles for us at the desired voxel size. | ||
| + | |||
| + | A script ''dynamo2warp'' is provided in the [[https://github.com/alisterburt/dynamo2m dynamo2m]] package. It requires a [[Table | table]] and a [ Tomogram-table map file | tomogram-table map file] and produces a STAR file with all the necessary information. | ||
| + | |||
| + | Particles are then extracted using the [[http://www.warpem.com/warp/?page_id=169 export subtomograms]] task dialogue in Warp. | ||
= run alignments/classifications in dynamo or relion = | = run alignments/classifications in dynamo or relion = | ||
| + | When warp exports subtomograms it provides a STAR file for the output. This can be used to run refinements in RELION. | ||
| + | |||
| + | If we want to import particles back into Dynamo, we run the ''warp2dynamo'' script from [https://github.com/alisterburt/dynamo2m dynamo2m] on this STAR file. | ||
| + | |||
| + | This script creates a table file and corresponding tomogram-table map file, this table file should be used for refinements with the Warp particles. | ||
| + | |||
| + | Due to some normalisation differences, it is better to create a dynamo formatted data folder from these particles. Two extra files are provided for this, with the suffix ''_reextract'' in their names. | ||
| + | |||
| + | A valid data folder can be cropped from these particles using the command | ||
| + | |||
| + | <nowiki>dtcrop <tomogram_table_map_reextract.doc> <table_reextract.tbl> <outputfolder> <sidelength> -asBoxes 1</nowiki> | ||
| + | |||
| + | You then have a valid data folder and table file with which you can start refinements. If you want to perform subboxing or reextract particles for whatever reason, simply use the ''dynamo2warp'' script on the correct table and tomogram-table map file at any time and reextract in Warp. | ||
= run multi-particle refinements in M = | = run multi-particle refinements in M = | ||
| + | M expects two independently refined half-maps, a mask and a corresponding ''_data.star'' file, as well as the settings file from the directory you used to process your frames in Warp. | ||
| + | |||
| + | We assume here that you have run a [[Adaptive_bandpass_filtering | gold standard]] alignment project in Dynamo, and thus have two tables and two half maps in the results of your alignment project. | ||
| + | |||
| + | Run the ''dynamo2m'' script in this directory, providing the two tables and the corresponding [[tomogram-table map file]]. This will give you a ''_data.star'' file which can be used in M. | ||
| + | |||
| + | Then, follow the guide for running [[http://www.warpem.com/warp/?page_id=827 multi-paricle refinements]] in M | ||
| + | |||
| + | = rerun alignments/classifications in dynamo or relion (optional) = | ||
| + | |||
| + | If you want to re-refine or re-classify particles after running multi-particle refinements in M export your particles from within M directly as described [[http://www.warpem.com/warp/?page_id=1631 here]] | ||
| + | |||
| + | If you would like to import these particles into dynamo please use the ''m2dynamo'' script from [[https://github.com/alisterburt/dynamo2m dynamo2m]] | ||
Latest revision as of 12:47, 19 August 2020
This page describes a method for integration of Dynamo, Warp and M to be able to take advantage of...
Warp's
- streamlined preprocessing
- robust CTF estimation procedures for tilt-series data
Dynamo's
- automated fiducial based tilt-series alignment
- geometric picking tools
- flexible subtomogram averaging procedures
M's
- multi-particle refinement
- half-map regularisation by denoising
Contents
Overview
- Preprocess from multi-frame micrographs to tilt-series using Warp
- Align tilt-series using [Dynamo]
- Generate downscaled tomograms in warp
- Pick particles in Dynamo and run first alignments
- Reextract unbinned particles in warp using [dynamo2m]
- Run alignments/classifications in dynamo or relion
- Run multi-particle refinements in M
- (optional) Rerun alignments/classifications in dynamo or relion
The last steps can be repeated to further improve final reconstruction quality
preprocess
Preprocess your multi-frame micrographs in warp following the guide [here]
Bad images should be deselected in Warp at this stage.
This stage is finished once you have your tilt-series in the imod folder
If you collected your data in Tomo rather than SerialEM and don't have mdoc files, you can try [mdocspoofer]
align tilt-series
Without Fiducials
Tilt-series without fiducials should be aligned in IMOD with patch-tracking
With Fiducials
Tilt-series can either be aligned in IMOD or using dynamo's new tilt series alignment tools
A function to automate the alignment of datasets with fiducials and convert all necessary metadata for import into warp is provided as autoalign_dynamo.
Once you've followed the setup from that page, simply navigate to the imod directory generated by warp and run
dautoalign4warp(<pixel_size_angstrom>, <fiducial_diameter_nm>, <nominal_rotation_angle>, <output_folder>)
The output folder is a folder which will contain all of your alignments for this dataset.
Briefly, tilt-series are aligned in Dynamo with the automated procedure. The fiducial positions are then converted into an IMOD model file and tiltalign is run on this to calculate the necessary transforms to apply to the tilt series in the format Warp wants.
generate downscaled tomograms
Tomograms are reconstructed at a larger pixel size in Warp for picking using the [tomography task dialogue] in Warp.
Choose to also generate a deconvolved version, we will use this for visualisation.
Generate a catalogue using the function warp2catalogue()provided in [autoalign_dynamo]
warp2catalogue(<warp_reconstruction_folder>, <pixel_size_angstrom>)
pick particles in dynamo
Pick particles using your favourite type of model through the catalogue created in the previous step.
Make sure to crop particles from the catalogue directly, this will make sure that even though you picked particles on filtered volumes for visualisation the particles are cropped from the unfiltered tomograms.
You can choose to run first sets of alignments on these downscaled particles to find which particles you would like to use for later refinements.
reextract unbinned particles in warp
Once you have a set of positions and orientations you want to use in refinements at a smaller pixel size, we need to convert this metadata into something Warp can understand so it can reconstruct our particles for us at the desired voxel size.
A script dynamo2warp is provided in the [dynamo2m] package. It requires a table and a [ Tomogram-table map file | tomogram-table map file] and produces a STAR file with all the necessary information.
Particles are then extracted using the [export subtomograms] task dialogue in Warp.
run alignments/classifications in dynamo or relion
When warp exports subtomograms it provides a STAR file for the output. This can be used to run refinements in RELION.
If we want to import particles back into Dynamo, we run the warp2dynamo script from dynamo2m on this STAR file.
This script creates a table file and corresponding tomogram-table map file, this table file should be used for refinements with the Warp particles.
Due to some normalisation differences, it is better to create a dynamo formatted data folder from these particles. Two extra files are provided for this, with the suffix _reextract in their names.
A valid data folder can be cropped from these particles using the command
dtcrop <tomogram_table_map_reextract.doc> <table_reextract.tbl> <outputfolder> <sidelength> -asBoxes 1
You then have a valid data folder and table file with which you can start refinements. If you want to perform subboxing or reextract particles for whatever reason, simply use the dynamo2warp script on the correct table and tomogram-table map file at any time and reextract in Warp.
run multi-particle refinements in M
M expects two independently refined half-maps, a mask and a corresponding _data.star file, as well as the settings file from the directory you used to process your frames in Warp.
We assume here that you have run a gold standard alignment project in Dynamo, and thus have two tables and two half maps in the results of your alignment project.
Run the dynamo2m script in this directory, providing the two tables and the corresponding tomogram-table map file. This will give you a _data.star file which can be used in M.
Then, follow the guide for running [multi-paricle refinements] in M
rerun alignments/classifications in dynamo or relion (optional)
If you want to re-refine or re-classify particles after running multi-particle refinements in M export your particles from within M directly as described [here]
If you would like to import these particles into dynamo please use the m2dynamo script from [dynamo2m]
