The smoothing mask is used in the context of adaptive bandpass filtering. This procedure needs an automatic evaluation of the attained resolution at each iteration step. This is performed through and FSC computation of the averages computed independently in the different channels of the odd/even computation.
Note that if one of the average of the channels is aligned to the other one (in order to create a comparison), this average will have a border alignment artifact resulting from the rotation and shift. If no masking is used, this would introduce an unwanted artifact on the high frequencies of the FSC. Thus, it is advisable to use a non-default (and smooth) mask for the resolution computation.
The minimal mask that we recommend is a sphere or the radius of the cube with with a falloff of two fourier pixels. When you run an adaptive bandpass filtering, one of the references need to be aligned to the other, which inevitably creates a sharp edge artifact in one of the two averages. Applying the same mask on both volumes will get rid of this effect.