The smoothing mask is used in the context of adaptive bandpass filtering. This procedure needs an automatic evaluation of the attained resolution at each iteration step. This is performed through and FSC computation of the averages computed independently in the different channels of the odd/even computation.
Note that if one of the average of the channels is aligned to the other one (in order to create a comparison), this average will have a border alignment artifact resulting from the rotation and shift. If no masking is used, this would introduce an unwanted artifact on the high frequencies of the FSC. Thus, it is advisable to use a non-default (and smooth) mask for the resolution computation.