Resolution, target density and labeling effects in colocalization studies - suppression of false positives by nanoscopy and modified algorithms

Colocalization analyses of fluorescence images are extensively used to quantify molecular interactions in cells. In recent years, fluorescence nanoscopy has approached resolutions close to molecular dimensions. However, the extent to which image resolution influences different colocalization estimates has not been systematically investigated. In this work, we applied simulations and resolution‐tunable stimulated emission depletion microscopy to evaluate how the resolution, molecular density and label size of targeted molecules influence estimates of the most commonly used colocalization algorithms (Pearson correlation coefficient, Manders’ M1 and M2 coefficients), as well as estimates by the image cross‐correlation spectroscopy method. We investigated the practically measureable extents of colocalization for stimulated emission depletion microscopy with positive and negative control samples with an aim to identifying the strengths and weaknesses of nanoscopic techniques for colocalization studies. At a typical optical resolution of a confocal microscope (200–300 nm), our results indicate that the extent of colocalization is typically overestimated by the tested algorithms, especially at high molecular densities. Only minor effects of this kind were observed at higher resolutions (< 60 nm). By contrast, underestimation of colocalization may occur if the resolution is close to the size of the label/affinity molecules themselves. To suppress false positives at confocal resolutions and high molecular densities, we introduce a statistical variant of Costes’ threshold searching algorithm, used in combination with correlation‐based methods like the Pearson coefficient and the image cross‐correlation spectroscopy approach, to set intensity thresholds separating background noise from signals