Difference between revisions of "Tips for management of tomographic data sets"

From Dynamo
Jump to navigation Jump to search
Line 20: Line 20:
 
<repository>/data/b002
 
<repository>/data/b002
 
<repository>/data/b015
 
<repository>/data/b015
 +
...
 
</nowiki>
 
</nowiki>
 +
Note that the batch numbers don't need to be consecutive.
  
==Tilt series folder===
+
===Tilt series folder===
 
+
Inside each batch folder we create a separate folder for each tilt series. The convention for the batch folder is a <tt>'ts'</tt> character followed with an integer (zero-padded to three figures), for instance
Inside each batch folder we create a separate folder for each tilt series. The convention for the batch folder is a <tt>'ts'</tt> character followed with an integer (zero-padded to three figures).
 
 
  <nowiki><repository>/data/b001/ts012
 
  <nowiki><repository>/data/b001/ts012
 
<repository>/data/b001/ts016
 
<repository>/data/b001/ts016
 +
...
 +
</nowiki>
 +
 +
Inside each ''tilt series folder'' we typically define the folders <tt>raw</tt> and <tt>imod</tt>
 +
<nowiki><repository>/data/bXXX/tYYY/raw
 +
<repository>/data/bXXX/tYYY/imod
 
</nowiki>
 
</nowiki>
  
Our acquisition system Focus delivers for each tilt series:
+
The contents of the <tt>raw</tt> folder may vary from project to project. They might be unaligned movies, gain reference files, or any intermediate files. Our acquisition system Focus delivers for each tilt series:
* a tilt series file, i.e., a stack file in .mrc format.  
+
* a tilt series file, i.e., a stack file in mrc format (and .mcrs extension).  
 
* a metadata file <tt>.star</tt> file.
 
* a metadata file <tt>.star</tt> file.
 
   
 
   
 
+
When we use <tt>Imod</tt> for reconstruction, we immediately transfer the stack to the <tt>imod</tt> folder. By convention we keep the batch and tilt series numbers and change the extension to <tt>.st</tt>
 
+
<nowiki><repository>/data/bXXX/tYYY/imod/bXXXtsYYY.st
 +
</nowiki>
  
 
==Creating the reconstructions==
 
==Creating the reconstructions==

Revision as of 12:05, 28 April 2017

The guidelines sketched here are a walkthrough on how we organize our tomographic data sets in the BioEM lab of the University of Basel.

We use a series of conventions on the way we storage of the raw data (i.e.) tilt series that makes easy to keep track on the processing steps already performed (alignment, reconstruction), the location of the intermediate results, and their binding into a Dynamo catalogue.

The raw data is stored in a tree below a repository folder, which will contain all the data (tilt series and tomograms) related to the tomography projects. The repository folder is created with the structure:


<repository>/data
<repository>/ctf

Here, <repository> is some location in your file system, probably a file share destined for massive storage (at least Tbs)

Organizing the tilt series stacks

Batches

We organize our tilt series in batches. Each batch represents customarily tilt series acquired in the same conditions or in the same session, but there is no strict rule. Each batch will have its own folder under the data repository folder. The convention for the batch folder is a 'b' character followed with an integer (zero-padded to three figures).

<repository>/data/b001
<repository>/data/b002
<repository>/data/b015
...

Note that the batch numbers don't need to be consecutive.

Tilt series folder

Inside each batch folder we create a separate folder for each tilt series. The convention for the batch folder is a 'ts' character followed with an integer (zero-padded to three figures), for instance

<repository>/data/b001/ts012
<repository>/data/b001/ts016
...

Inside each tilt series folder we typically define the folders raw and imod

<repository>/data/bXXX/tYYY/raw
<repository>/data/bXXX/tYYY/imod

The contents of the raw folder may vary from project to project. They might be unaligned movies, gain reference files, or any intermediate files. Our acquisition system Focus delivers for each tilt series:

  • a tilt series file, i.e., a stack file in mrc format (and .mcrs extension).
  • a metadata file .star file.

When we use Imod for reconstruction, we immediately transfer the stack to the imod folder. By convention we keep the batch and tilt series numbers and change the extension to .st

<repository>/data/bXXX/tYYY/imod/bXXXtsYYY.st

Creating the reconstructions

Creating the first catalogue

Prebinning of catalogue volumes

Visualizing the tomograms