Quantifying suppression of the cosmological 21-cm signal due to direction dependent gain calibration in radio interferometers

The 21-cm signal of neutral hydrogen promises to be an important source of information for the study of the cosmic dawn and epoch of reionization. However, its detection is difficult without sufficient mitigation of strong contaminating signals in the data, requiring accurate instrument gain calibration. To detect the 21-cm signal, one analyses the resulting post-calibration residual data. The 21-cm signal residing in these residuals, however, can be strongly affected by the process of direction-dependent calibration (DDC) combined with sky-model removal. The impact of this, in particular on the frequency behaviour of the 21-cm signal, has remained poorly quantified. In this paper, we derive a new mathematical formalism on how to calculate the suppression of the 21-cm signal during DDC and sky-model removal. We also show how a priori information about the frequency behaviour of the instrument, such as spectral smoothness, can be utilized to reduce signal suppression. The theoretical results are tested using a realistic simulation based on the LOFAR-EoR calibration set-up. The results show that if the instrumental gains are intrinsically smooth over several MHz (e.g. polynomials over a bandwidth of similar to 10 MHz), this is sufficient to allow for DDC in as many as similar to 100 directions with limited and quantifiable suppression of the 21-cm signal power spectrum. We also demonstrate that more incomplete sky models during calibration lead to larger 21-cm signal suppression, even if the instrumental gain models are fully smooth. This result has immediate consequences for future radio telescopes with non-identical station beams, where DDC might be necessary (e.g. SKA-Low)