Tips for management of tomographic data sets

From Dynamo
Revision as of 15:37, 28 April 2017 by Daniel Castaño (talk | contribs)
Jump to navigation Jump to search

The guidelines sketched here are a walkthrough on how we organize our tomographic data sets in the BioEM lab of the University of Basel.

We use a series of conventions for the storage of the raw data (i.e.) tilt series, in order to make easy to keep track on the processing steps already performed (alignment, reconstruction), the location of the intermediate results, and their binding into a Dynamo catalogue. The idea is to keep and uniform naming convention that allows to comfortably locate the sought files and folders with simple Linux commands.

In this example, we assume that you are using imod for tomographic alignment and reconstruction of each tilt series. The raw data is stored in a tree below a repository folder, which will contain all the data (tilt series and tomograms) related to the tomography projects. The repository folder is created with the structure:

<repository>/data
<repository>/ctf

Here, <repository> is some location in your file system, probably a file share destined for massive storage (at least Tbs). We often call this location 'repository' (literally).

Organizing the tilt series stacks

Batches

We organize our tilt series in batches. Each batch represents customarily tilt series acquired in the same conditions or in the same session, but there is no strict rule. Each batch will have its own folder under the data repository folder. The convention for the batch folder is a 'b' character followed with an integer (zero-padded to three figures).

<repository>/data/b001
<repository>/data/b002
<repository>/data/b015
...

Note that the batch numbers don't need to be consecutive.

Tilt series folder

Inside each batch folder we create a separate folder for each tilt series. The convention for the batch folder is a 'ts' character followed with an integer (zero-padded to three figures), for instance

<repository>/data/b001/ts012
<repository>/data/b001/ts016
...

Inside each tilt series folder we typically define the folders raw and imod

<repository>/data/bXXX/tsYYY/raw
<repository>/data/bXXX/tsYYY/imod

The contents of the raw folder may vary from project to project. They might be unaligned movies, gain reference files, or any intermediate files. Our acquisition system Focus delivers for each tilt series:

  • a tilt series file, i.e., a stack file in mrc format (and .mcrs extension).
  • a metadata file .star file.
  • a .tlt file with the values of the tilt angles of each micrograph contained in the stack.

When we use Imod for reconstruction, we immediately transfer the stack to the imod folder. By convention we keep the batch and tilt series numbers and change the extension to .st

<repository>/data/b002/ts013/imod/b002ts013.st

Creating the reconstructions

After using etomo in the folder that contains the tilt series, you should end up with a full sized tomogram in the corresponding imod folder. We name this tomogram bXXXtsYYYFull.mrc <repository>/data/b002/ts013/imod/b002ts013Full.mrc

Reconstruction Post processing

Importantly, we usually want to have tomograms in which z represents the direction of the electron beam and y represents the direction of the tilt axis. This usually require flipping the tomogram. You can do it already in the etomo GUI, or with the imod command line, or with Dynamo commands or later in the Dynamo GUI.

In this walkthrough, let's assume that the files

<repository>/data/bYYY/tsYYY/imod/bXXXtsYYYFull.mrc

have already been flipped before entering them into the Dynamo catalogue that we will generate later.

Creating a catalogue

With the convention that we have used so far, you can use in Linux the order

ls -d <repository>/data/b*/ts*/imod/*Full.rec

to list into the screen your available tomograms, each with its absolute path.

Then, in order to create a catalogue you just need to create a text file with them

ls -d <repository>/data/b*/ts*/imod/*Full.rec >> list.vll 

The list.vll we just created is the most basic syntax of a volume list file, but we can already use it to generate a catalogue of tomograms. Inside Matlab (or the standalone), just type

dcm -create myCatalogue -fromvll list.vll

and a catalogue will be generated in the current location.

You can open it inside Matlab/Dynamo with

dcm -c myCatalogue

Prebinning of catalogued tomograms

The tomograms are now linked to the catalogue. They are however too big to work with them comfortably. They take a long time to be read, and then they consume a lot of memory. We thus always prebind themPrebinned tomograms, usually with a factor of two. Thus, tomograms with size 4k x 4k x 1k will be visualized through a precomputed proxy with a size of 1k x 1k x 250 pixels.

The full procedure takes typically some minutes per tomogram.

The proxy tomograms are stored as files with name bXXXtsYYYFull_CatBinned2.mrc. You don't need to keep track of them. Whenever you select the original full size tomogram for visualization purposes, Dynamo will check for the presence of a prebinned one and offer to open it instead.

Visualizing the tomogram set

In principle you're ready to go and could open a single tomogram to start creating

Gallery of tomograms

File:.png
Opening the live gallery option
File:.png
Options in the live gallery
Thumbnails

Thumbnails are very binned versions of the original tomograms, useful to get a quick view of the contents of each them.

In order to create them, it's convenient to have already ran the prebinning procedure (as Dynamo can then work directly on the prebinned versions, and the creation of the thumbnail tomograms will then take just a few seconds)

File:.png
Creating the thumbnails and accessing them
File:.png
The thumbnail gallery



Accessing each tomogram

You need to access each tomogram individually from the main catalogue GUI.