The alignment mask is use during alignment. The template is rotated and shifted (virtually, through fourier acceleration). At each posible combination of rotation angles and shift, the mask is also rotated and shifted, defining a moving region inside the template. The rotated and shifted template is compared to the data particle only inside this moving region.
There is no need to define a smooth mask: the mask is never applied onto the template (never gets pixelwise multiplied against the template). The alignment mask is rather used to define the region inside which template and particle will be compared.
This moving region is never produced explicitly: the effect of selecting a moving area is coded inside the Fourier acceleration.
Alignment mask during averaging
The alignment mask is used once again during the averaging step. After rotating and shifting the particle along the alignment parameters, the full particle is normalized to zero mean and standard deviation one inside the region defined by the alignment mask. No further use of the alignment mask is performed: not for FSC computations, or filtering.
Alan M. Roseman, Particle finding in electron micrographs using a fast local correlation algorithm, Ultramicroscopy 94 (2003) 225–236