Tips for management of tomographic data sets

The guidelines sketched here are a walkthrough on how we organize our tomographic data sets in the BioEM lab of the University of Basel.

We use a series of conventions on the way we storage of the raw data (i.e.) tilt series that makes easy to keep track on the processing steps already performed (alignment, reconstruction), the location of the intermediate results, and their binding into a Dynamo catalogue.

The raw data is stored in a tree below a repository folder, which will contain all the data (tilt series and tomograms) related to the tomography projects. The repository folder is created with the structure:

<repository>/data
<repository>/ctf

Here, <repository> is some location in your file system, probably a file share destined for massive storage (at least Tbs)

Organizing the tilt series stacks

Batches

We organize our tilt series in batches. Each batch represents customarily tilt series acquired in the same conditions or in the same session, but there is no strict rule. Each batch will have its own folder under the data repository folder. The convention for the batch folder is a 'b' character followed with an integer (zero-padded to three figures).

<repository>/data/b001
<repository>/data/b002
<repository>/data/b015
...

Note that the batch numbers don't need to be consecutive.

Tilt series folder

Inside each batch folder we create a separate folder for each tilt series. The convention for the batch folder is a 'ts' character followed with an integer (zero-padded to three figures), for instance

<repository>/data/b001/ts012
<repository>/data/b001/ts016
...

Inside each tilt series folder we typically define the folders raw and imod

<repository>/data/bXXX/tYYY/raw
<repository>/data/bXXX/tYYY/imod

The contents of the raw folder may vary from project to project. They might be unaligned movies, gain reference files, or any intermediate files. Our acquisition system Focus delivers for each tilt series:

• a tilt series file, i.e., a stack file in mrc format (and .mcrs extension).
• a metadata file .star file.

When we use Imod for reconstruction, we immediately transfer the stack to the imod folder. By convention we keep the batch and tilt series numbers and change the extension to .st

<repository>/data/bXXX/tYYY/imod/bXXXtsYYY.st