An Improved Joint Optimization of Multiple Level Set Functions for the Segmentation of Overlapping

Abstract—In this paper we present an improved algorithm for the segmentation of cytoplasm and nuclei from clumps of overlapping cervical cells. This problem is notoriously difficult because of the degree of overlap among cells, the poor con-trast of cell cytoplasm and the presence of mucus, blood and inflammatory cells. Our methodology addresses these issues by utilising a joint optimization of multiple level set functions, where each function represents a cell within a clump, that have both unary (intra-cell) and pairwise (inter-cell) constraints. The unary constraints are based on contour length, edge strength and cell shape, while the pairwise constraint is computed based on the area of the overlapping regions. In this way, our methodology enables the analysis of nuclei and cytoplasm from both free-lying and overlapping cells. We provide a systematic evaluation of our methodology using a database of over 900 images generated by synthetically overlapping images of free-lying cervical cells, where the number of cells within a clump is varied from 2 to 10 and the overlap coefficient between pairs of cells from 0.1 to 0.5. This quantitative assessment demonstrates that our methodology can successfully segment clumps of up to 10 cells, provided the overlap between pairs of cells is below 0.2. Moreover, if the clump consists of three or fewer cells, then our methodology can successfully segment individual cells even when the overlap is around 0.5. We also evaluate our approach quantitatively and qualitatively on a set of 16 extended depth of field images, where we are able to segment a total of 645 cells, of which only around 10 % are free-lying. Finally, we demonstrate that our method of cell nuclei segmentation is competitive when compared to the current state of the art. Index Terms—Overlapping cell segmentation, Pap smear image analysis, Level set method I