View source for Tips for management of tomographic data sets


The guidelines sketched here are a walkthrough on how we organize our tomographic data sets in the BioEM lab of the University of Basel.   

We use a series of conventions for the storage of the raw data (i.e.) tilt series, in order to make easy to keep track on the processing steps already performed (alignment, reconstruction), the location of the intermediate results, and their binding into a ''Dynamo'' [[catalogue]]. The idea is to keep and uniform naming convention that allows to comfortably locate the sought files and folders with simple Linux commands.  

In this example, we assume that you are using <tt>imod</tt> for tomographic alignment and reconstruction of each tilt series. The raw data is stored in a tree below a ''repository'' folder, which will contain all the data (tilt series and tomograms) related to the tomography projects.  The repository folder is created with the structure:

 <nowiki><repository>/data
<repository>/ctf</nowiki>

Here, <tt><repository></tt> is some location in your file system, probably a file share destined for massive storage (at least Tbs). We often call this location <tt>'repository'</tt> (literally).

== Organizing the tilt series stacks==

=== Batches ===
We organize our tilt series in batches. Each batch represents customarily tilt series acquired in the same conditions or in the same session, but there is no strict rule. Each batch will have its own folder under the <tt>data</tt> repository folder. The convention for the batch folder is a <tt>'b'</tt> character followed with an integer (zero-padded to three figures).
 <nowiki><repository>/data/b001
<repository>/data/b002
<repository>/data/b015
...</nowiki>
Note that the batch numbers don't need to be consecutive.

===Tilt series folder===
Inside each batch folder we create a separate folder for each tilt series. The convention for the batch folder is a <tt>'ts'</tt> character followed with an integer (zero-padded to three figures), for instance
 <nowiki><repository>/data/b001/ts012
<repository>/data/b001/ts016
...</nowiki>

Inside each ''tilt series folder'' we typically define the folders <tt>raw</tt> and <tt>imod</tt>
 <nowiki><repository>/data/bXXX/tsYYY/raw
<repository>/data/bXXX/tsYYY/imod
</nowiki>

The contents of the <tt>raw</tt> folder may vary from project to project. They might be unaligned movies, gain reference files, or any intermediate files. Our acquisition system Focus delivers for each tilt series:
* a tilt series file, i.e., a stack file in mrc format (and .mcrs extension). 
* a metadata file <tt>.star</tt> file. 
* a <tt>.tlt</tt> file with the values of the tilt angles of each micrograph contained in the stack.
 
When we use <tt>Imod</tt> for reconstruction, we immediately transfer the stack to the <tt>imod</tt> folder. By convention we keep the batch and tilt series numbers and change the extension to <tt>.st</tt> 
 <nowiki><repository>/data/b002/ts013/imod/b002ts013.st</nowiki>

==Creating the reconstructions==

After using <tt>etomo</tt> in the folder that contains the tilt series, you should end up with a full sized tomogram in the corresponding <tt>imod</tt> folder. We name  this tomogram <tt>bXXXtsYYYFull.mrc</tt>
<nowiki><repository>/data/b002/ts013/imod/b002ts013Full.mrc</nowiki>

=== Reconstruction Post processing===
Importantly, we usually want to have tomograms in which ''z'' represents the direction of the electron beam and ''y'' represents the direction of the tilt axis. This usually require ''flipping'' the tomogram. You can do it already in the <tt>etomo</tt> GUI, or with the imod command line, or with [[Operations on volumes|''Dynamo'' commands]] or later in the ''Dynamo'' GUI.

In this walkthrough, let's assume that the files
 
<nowiki><repository>/data/bYYY/tsYYY/imod/bXXXtsYYYFull.mrc</nowiki>

have already been flipped before entering them into the ''Dynamo'' [[catalogue]] that we will generate later.

==Creating a catalogue== 

With the convention that we have used so far, you can use in Linux  the order
 <tt>ls -d <repository>/data/b*/ts*/imod/*Full.rec</tt>
to list into the screen your available tomograms, each with its absolute path.

Then, in order to create a catalogue you just need to create a text file with them
 <tt>ls -d <repository>/data/b*/ts*/imod/*Full.rec >> list.vll </tt>

The <tt>list.vll</tt> we just created is the most basic syntax of a [[volume list file]], but we can already use it to generate a catalogue of tomograms. Inside Matlab (or the [[standalone]]), just type 

 <tt>dcm -create myCatalogue -fromvll list.vll</tt>

and a [[catalogue]] will be generated in the current location.

You can open it inside Matlab/Dynamo with

 <tt>dcm -c myCatalogue</tt>

===Prebinning of catalogued tomograms===

The tomograms are now linked to the catalogue. They are however too big to work with them comfortably. They take a long time to be read, and then they consume a lot of memory. We thus always [[prebind themPrebinned tomograms]], usually with a factor of two. Thus, tomograms with size 4k x 4k x 1k will be visualized through a precomputed proxy with a size of 1k x 1k x 250 pixels.

The full procedure takes typically some minutes per tomogram.

The proxy tomograms are stored as files with name bXXXtsYYYFull_CatBinned2.mrc. You don't need to keep track of them. Whenever you select the original full size tomogram for visualization purposes, ''Dynamo'' will check for the presence of a prebinned one and offer to open it instead.

===Visualizing the tomogram set===

In principle you're ready to go and could open a single tomogram to start creating annotations. Remember however that the [[dcm GUI]] includes a couple of tools for quickly checking all the tomograms in a catalogue.

==== Live gallery of tomograms ====
The ''live gallery'' shows one projection of several slices from each of the tomograms. The slices are read on the fly,i.e., only the part of the tomogram needed for creating the depicted scene will be read.
[[File:RepositoryGalleryMenu.png|thumb|center|400px|Opening the live gallery option]]

After opening the live gallery you are most likely to need to adjust depiction parameters as binning level, bandpass or number of projected slices per tomogram. 
[[File:RepositoryGalleryOptions.png|thumb|center|800px|Options in the live gallery]]

===== Thumbnails =====
Thumbnails are very binned versions of the original tomograms, useful to get a quick view of the contents of each them. 

In order to create them, it's convenient to have already ran the prebinning procedure (as ''Dynamo'' can then work directly on the prebinned versions, and the creation of the thumbnail tomograms will then take just a few seconds) 

[[File:RepositoryThumbnailsCreate.png|thumb|center|300px|Creating the thumbnails and accessing them]]
[[File:RepositoryThumbnailsGallery.png|thumb|center|800px|The thumbnail gallery]]

=== Accessing each tomogram ===

==== Check on the fly ====


==== Ready to work ====
When you are ready to start working on a tomogram (complex visualziation, model definition, annotation, etc), you want to open it with [[dtmslice]]. To do so, the best way is to secondary click on the row of the tomogram and use the option of sending the tomogram into [[dtmslice]]. If you have created some binned versions, they will appear as available to be opened. Always '''use the prebinned tomograms''' if the original ones are too big!

[[File:RepositoryTomoslice.png|thumb|center|800px|Selecting a tomogram and opening its prebinned version in [[dtmslice]]]]