Table

Tables are the basical metadata system in Dynamo. They are just matrices that describe the properties of sets of particles. The property coded on each column is fixed.

A table can be seen as the metadata of a given data folder. The first column in a table is just an integer number (which we call a tag) that identifies a particle. In the data folder, this particle is identified by the name of the file that contains it, i.e.

<myData>/particle_00003.em

would be the particle with tag 3 in the data folder myData. A table will be able to operate on this particle if it has a row whose first column is the integer 3.

As Tables are matrices, Matlab tools (like find,plot,hist) are easily customizable to explore and visualize the contents of a table. Additionally, Dynamo includes many specific utilities. dapropos table will give you a list of basic commands that you can use for table manipulation.

Uses of table objects

Inside Dynamo, tables appear in mainly three scenarios. The [#Format convention in a table|formatting convention] is however always the same.

Initial metadata for alignment projects

Assumed to code the initial shifts and angles of the particles (if known a priori)

Results of alignment iterations

They are automatically produced by Dynamo at the end of each iteration of a project.

Extracting particles from volumes

Main article: Cropping table

Any table can be used to extract particles from a volume or set of volumes. The spacial coordinates need to be coded in columns 24 to 26. The same table can be used for several volumes when used jointly with a volume-table mapping file.

Format convention in a table

Main article: Table column convention

Rows correspond to particles, columns correspond to predefined properties. The convention used in the definition of each column of the table can be consulted each time by the command dthelp

1   : tag           tag of particle file in data folder
2   : aligned       value 1: marks the particle for alignment
3   : averaged      value 1: the particle was included in the average
4   : dx            x shift from center (in pixels)
5   : dy            y shift from center (in pixels)
6   : dz            z shift from center (in pixels)
7   : tdrot         euler angle (rotation around z, in degrees)
8   : tilt          euler angle (rotation around new x, in degrees)
9   : narot         euler angle (rotation around new z, in degrees)
10  : cc            Cross correlation coefficient
11  : cc2           Cross correlation coefficient after thresholding II
12  : cpu           processor that aligned the particle
13  : ftype         0: full range; 1: tilt around y ( ... dhelp dtutorial for more options)
14  : ymintilt      minimum angle in the tilt series around tilt axis (i.e. -60)
15  : ymaxtilt      maximum angle in the tilt series around tilt axis (i.e. 60)
16  : xmintilt      minimum angle in the tilt series around second tilt axis (i.e. -60)
17  : xmaxtilt      maximum angle in the tilt series around second x (i.e. 60)
18  : fs1           free parameter for fourier sampling #1()
19  : fs2           free parameter for fourier sampling #2()
20  : tomo          tomogram number 
21  : reg           for arbitrary assignations of regions inside tomograms
22  : class         class number 
23  : annotation    use this field for assigning arbitrary labels
24  : x             x coordinate in original volume
25  : y             y coordinate in original volume
26  : z             z coordinate in original volume
27  : dshift        norm of shift vector
28  : daxis         difference in axis orientation
29  : dnarot        difference in narot
30  : dcc           difference in CC
31  : otag          original tag (subboxing)
32  : npar          number of added particles (compactions) / subunit label (subboxing)
34  : ref           reference. Used in multireference projects
35  : sref          subreference (i.e. generated by Dynamo PCA)
36  : apix          angstrom per pixel
37  : def           defocus (in micron)
41  : eig1          "eigencoefficient" #1
42  : eig2          "eigencoefficient" #2
 
more columns can be added for user-defined properties.

Reading and writing tables

Tables are customarily stored as text files with extension '.tbl'.

Tables can be read and write into the matlab worspace with the standard commands dwrite and dread.

For specially long tables, you can use the extension '.tbl', which will write the files in binary format, allowing considerable speedup in I/O.

Visualization of tables

Basic info

A summary of contents in a table can be displayed by dtinfo. Typing the command:

dinfo mytable.tbl

on a table file mytable.tbl you will get the mean and std values for the contents of each column of the table.

Plots through command line

dtplot allows the creation of graphs like: positions of particles in a volume, orientations, distributions of orientations. For instace, you can check if all the particles in a reference (column = 34) have a similar missing wedge orientation (the tilt angle is coded in column 8).

For instance:

dtplot example.tbl -pf oriented_positions

shows the locations and orientations of all the particles in a table.

dtplot used with profile "oriented positions"

Basic plots through GUI

The basic GUI is dtshow. Typing:

dtshow file.tbl

for a file containing a table, or

dtshow(myTable)

for a table defined in the workspace, opens a lightweight GUI that allows quick plotting of the contents of the table, and also a GUI access to all the plotting options of dtplot. This GUI is one of options that the dcp GUI offers when exploring the results of an alignment through browsing the generated tables.

GUI for complex plots dtview

dtview is a more complicated browser for table contents, allowing the joint visualization of pairs of properties for particle sets inside a table.

Table manipulation

Creating tables

In general, Dynamo itself creates correctly formatted tables as they are needed. If a table needs to be explicitly created, an easy way to create a blank table compatible with a given data folder is with the command dynamo_table_blank.

Selection of parts of a table

Use dtgrep. It's applicability ranges from just selecting an explicitely defined range of particle tags to the computation of particles pointing in a direction.

Merging tables

Low level manipulation

Tables are just matrices. As such, they can be easily merged with simple Matlab manipulation tools. For instance, to merge the tables in files table file 1 and table file 2 and set the result into file merged table file, you just read the contents into the workspace and use the cat commnad for matrix concatenation:

t1 = dread(<table file 1>); 
t2 = dread(<table file 2>);
tMerged = cat(1,t1,t2); % concatenate rows of t1 and t2
dwrite(tMerged,<merged table file>);

When you merge tables manually, please make sure than the merged tables do not have overlapping tags (column 1).

You can check this by displaying the result of the operation.

 length(unique(tMerged(:,1)))

Merging commands

Dynamo also includes a dynamo_table_merge command, available from Matalb or the standalone

Conversion tools for other software packages

dapropos compatibility will print a selection of functions that can be used to import and export particle metadata with the format expected by other packages, as AV3, jsubtomo and XMIPP