Programmatic control of alignment and reconstruction workflows

Article in progress. These functionalities will be released in the Dynamo 1.2 series.

Alignment and reconstruction workflows can be used with the GUI or on a totally programmatic manner. Programmatic control allows for batched reconstructions of sets of several tilt series.

Description

This section describes qualitatively the organization logic of the workflow and it elements. Actual syntax is described in the next section

The workflow folder

Workflows in Dynamo are stored typically in folders (named with a capital case extension) that contain:

• a file that represents the workflow object itself. You kind think of it as a lightweight database that keeps track of every operation and parameter value.
• files and folders containing intermediate results.

Alignment and reconstruction workflow folders are marked with the extension .AWF

The workflow object

The workflow object is contained in a file called object.mat inside the .AWF folder.

If you need to perform low-level operations on the object, your can bring it to memory through dread

w = dread(<my workflow folder>);

or

w = dread(<my workflow folder>/object.mat);

which will create a memory a variable w that represents the workflow object. Most users, however, do not need to operate directly on the object, and can use instead

• dtsa for invoking a GUI that controls a workflow and allows on-screen design of steps and parameter selection.
• dtsar (dynamo_tilt_series_alignment_run) for running a workflow, i.e. executing all or part of their steps in non-interactive mode.

Both commands are valid for creating new workflows or manipulating existing ones.

Results

By default, intermediate and final results will be stored as fixed locations inside the workflow folder. This is the recommended behaviour for starting users.

Input data

Data does not need to be stored physically in the workflow folder.

Tilt series data

The full sized tilt series can be stored somewhere else and just linked to the workflow, as mrc or st file.

Original location of the tilt series

If you lose the overview of where the tilt series in a workflow is actually located, you can use general data location system of the workflow (explained elsewhere), or just the shortcut:

file = w.io.getMatrixFile()

where w is a workflow object.

Tilt angles

By default, the nominal tilt angles as delivered by the data acquisition software will be stored in the workflow folder, as file name nominalTiltAngles.tlt which is a text file. The order of the angles must correspond to order of tilt series stack.

Manually excluded tilt indices

Some micrographs might be detected to be of low quality by simple visual inspection (because of focusing errors, big shifts, etc). The identity of such micrographs is stored in a text file called discardedTiltIndices.txt. This file is updated when the workflow is handled through a GUI and the user excludes tilts manually. On the other hand, if the identity of the excluded tilts is known, the user can simply create a text file with the affected indices in the pre-established location.

Indices are always referred to the angle ordering of nominalTiltAngles.tlt

Algorithmically excluded tilt indices

Bear in mind that further micrographs in a tilt series might be excluded during the alignment procedure, even if the user didn't mark them for forceful exclusion. This happens when the gold bead indexing algorithm fails to recognize the identity of all the markers of a micrograph, or if the fitting error computed on the markers of a micrograph is considered too high. These indices will be stored under the file dynamicallyExcludedTiltIndices.txt

Syntax

Command line acces to workflows can be used in different ways:

• Creating a workflow for posterior use.
• Accessing a workflow previously created
• Creation and execution: all the way from tilt series file to reconstructed tomogram files.

Create and run workflow

The minimal input that you would need to create a reconstruction from a raw tilt series would be:

• an file containing the stack of micrographs (arbitrarily called stack.mrc) , and
• a text file containing the nominal tilt angles of each one of the micrographs in the stack (arbitrarily called angles.tlt) .

Thus, a command like:

dtsar -c myTest  -ts stack.mrc -nta  angles.tlt 

will:

• create the workflow folder myTest.AWF, with its corresponding workflow object,
• link the to the workflow object the position of the tilt series anf tilt angles,
• run all the tasks of gold detection, trace indexing, stack alignment and reconstruction using default parameters, and
• store all the (intermediate and final) results in the workflow folder.

This is the minimal set of instructions that will go all the way from the raw data to down to a reconstruction.

Tuning the input data

Manually excluded tilts

If you know beforehand that some tilts represent low quality or unusable information, you can pass it to the dtsar command through the mdt flag:

dtsar -c myTest  -ts stack.mrc -nta  angles.tlt -mdt excludedIndices.txt 

Note that the indices in the text tile excludedIndices.txt will be read and used to generate the workflow file myTest.AWF/excludedTiltIndices. It means that posterior edition of the file that you entered originally excludedIndices.txt will not change the behaviour of the workflow regarding excluding tilts. To exclude further tilts, you will have to

• edit the text file in the workflow folder myTest.AWF/excludedTiltIndices, or
• change manually the excluded tilt indices in the GUI of the workflow.

Tuning the tasks

By default, dtsar will drive the workflow to complete all its tasks: gold bead detection, trace indexing, trace refinement, stack alignment, and tomogram reconstruction. You might want to carry only a subset of tasks using the flag '-tasks', followed by the a letter that identifies the task with this convention:

• gold bead detection
• trace indexing
• trace refinement
• stack alignment
• tomogram reconstruction

Thus, the command

dtsar -c wf -ts file.mrc -nta tilts.txt -tasks difa

will skip the reconstruction task

Create a workflow for later use

All the previous syntax options can used with the addition of the flag -run 0. In such cases, the workflow will just be created but not executed.

Running preexisting workflow

Existing workflows can be re-run by replacing the flag -c (create) with -o (open). For instance, the command:

dtsar -o oldTest 

will open the workflow oldTest and execute all the tasks with the parameters found in disk. If the workflow is not found in disk, the command will not create a new one but issue an error instead. The previous order it will basically recompute and overwrite the previous contents. However, the previous syntax options are still valid, so that rerunning a workflow alignment with a new set of parameters can be performed through:

dtsar -o oldTest -pf newSet.param

Here

Use for batching

Recommended approach

Use through the repository system