Cropping unbinned particles after having worked with binned particles
Work in progress You will do a lot of operations with binned data before moving back to your full resolution data.
Contents
Basic approach
The simplest approach consists in:
- Keeping one catalogue to crop particles in full resolution.
- Creating binned versions of the tomograms for visualization purposes.
- When particles need to be binned to ease computations, a binning order is passed to the alignment or classification project.
This will work in many situations. Sometimes, however, cropping particles in full resolution can be tedious or prohibitively expensive. When operating seed oversampling for instance, most cropped particles will be excluded before the analysis comes to the point where full resolution pixels are needed.
Walkthrough on synthetic data
Creating a data set
This part of the walkthrough serves only to create a synthetic data set, comprising 3 full sized tomograms, each one containing already a model. Models are simple clouds of points.
Creates the catalogue in full size (let us assume 1 a.u. -arbitrary unit- per pixel)
dctutorial full -n 3 -cc withmodels
In real life, you are advised to always have a catalogue that links the "big", full resolution tomogram
Let Dynamo create a 1x binned version of each tomogram.
dynamo_catalogue_bin full_withmodels 1 -ss 256
-ss 256 defines the size of the chunks that are loaded at once from disk. This parameter does not matter for small tomograms like here. Each newly created tomogram will have a pixel size of 2 a.u.
Now, we create a new catalogue where the 1x bin versions are the main entries (meaning that the particles will be cropped from the binned version);
dcm -c full_withmodels -l t -ws o
Creates a list with the names of the binned tomograms
binnedFiles=strrep((o.results),'.mrc','_CatBinned1.mrc');
A good policy is to check that the chars that we generated correspond to files that actually exist:
isfile(binnedFiles);
should produce an array of 3 "trues", i.e:
% ans = % % 1×3 logical array % % 1 1 1
% % create the new catalogue % We creates a text file which just a list of files
mbio.writeas.textlines(binnedFiles,'binnedFileList.vll');
which is used to create a catalogue: dcm -create binned1 -fromvll binnedFileList.vll
To imports the models in the main catalogue we need to write a small script that loops on the preexisting models in full resolution in the full catalogue and creates corresponding models in the binned resolution catalogue. Loops are not easily handled from the command line, therefore we create a text file; a script that will run this task:
edit scaleModels
will open an editor where you can write:
nVolumes = 3;
myCatalogue = dread('binned1.ctlg');
lines = {}; % initialize a cell array
for i=1:nVolumes;
lines{end+1} = ['# just a comment'];
lines{end+1} = myCatalogue.volumes{i}.fullFileName();
% get the models in thid volume
modelsInVolume=myCatalogue.volumes{i}.modelFiles();
for imo=1:length(modelsInVolume);
lines{end+1} = ['> ',modelsInVolume{imo}];
end
lines{end+1} = ' ';
end
mbio.writeas.textlines(lines,'binned.vll');
We run the script just by writing its name in the command line:
scaleModels;
Checks that the new catalogue has indeed 3 models:
dcmodels binned1
Cropping particles from the binned tomogram
We need to creates a "vll" text file that expresses which models will come from which tomogram. The format is as follows:
tomogram1 >model file 1 in tomogram 1
Again, as this operaiton involves loops, we write an adhoc script:
edit createCroppingVll();
and fill it with the text:
nVolumes = 3;
myCatalogue = dread('binned1.ctlg');
lines = {}; % initialize a cell array
for i=1:nVolumes;
lines{end+1} = ['# just a comment'];
lines{end+1} = myCatalogue.volumes{i}.fullFileName();
% get the models in thid volume
modelsInVolume=myCatalogue.volumes{i}.modelFiles();
for imo=1:length(modelsInVolume);
lines{end+1} = ['> ',modelsInVolume{imo}];
end
lines{end+1} = ' ';
end
mbio.writeas.textlines(lines,'binned.vll');
We then use the script by:
createCroppingVll();
The created vll file can be used directly with dtcrop:
dtcrop binned.vll reorder targetData 24;
Here, the 'reorder' string as second argument tells dynamo to parse the contents of the vll and to use a new set of tags on the created particles.
To check the results
ddbrowse -d targetData -t targetData/crop.tbl
Cropping particles from the full resolution tomogram
Ok, now the actual "meat", i.e., the simulation of how we should proceed in real conditions. How do we crop the particles in the unbinned tomogram using as input the table collected on the binned particles? Remember, we disregard the metadata from the full size tomogram?
Upscale the coordinates in the table
We first upscale the table:
tUpscale = dynamo_table_rescale('targetData/crop.tbl','factor',2);
Tomogram map file
Then we create a text file that indicates which tomogram file is intended for which tomogram number in the table (remember that tomogram numbers are represented in the index entry in the column 20).
The text file is formatted as
% 1 tomogramFileForIndex1 % 2 tomogramFileForIndex2
We probably need to set the orders we need in a separate script:
edit createTomogramIndexMapFile
where you can write the loop on the file names.
binnedCatalogue = dread('binned1.ctlg');
linesMap = {};
foundIndices = tUpscale(:,20);
for i=1:length(foundIndices)
thisIndex = foundIndices(i);
fileBinned = binnedCatalogue.volumes{thisIndex}.fullFileName();
fileUnbinned = strrep(fileBinned,'_CatBinned1','');
if ~isfile(fileUnbinned)
error('Hm... the corresponding unbinned file cannot be found: %s',fileUnbinned);
end
linesMap{end+1} = [num2str(thisIndex), ' ',fileUnbinned];
%linesMap{end+1} = [num2str(i), ' ',fileUnbinned];
end
% writes the actual file
mbio.writeas.textlines(linesMap,'mapFullTomograms.doc');
We run the newly created script:
createTomogramIndexMapFile();
Actual cropping
Now we crop the particles with the newly created file mapFullTomograms
dtcrop('mapFullTomograms.doc',tUpscale,'targetDataFull', 48);
Now we check the results:
ddbrowse -d targetDataFull -t targetDataFull/crop.tbl
