Decoupling Zero-Shot Semantic Segmentation

Zero-shot semantic segmentation (ZS3) aims to segment the novel categoriesthat have not been seen in the training. Existing works formulate ZS3 as apixel-level zero-shot classification problem, and transfer semantic knowledgefrom seen classes to unseen ones with the help of language models pre-trainedonly with texts. While simple, the pixel-level ZS3 formulation shows thelimited capability to integrate vision-language models that are oftenpre-trained with image-text pairs and currently demonstrate great potential forvision tasks. Inspired by the observation that humans often performsegment-level semantic labeling, we propose to decouple the ZS3 into twosub-tasks: 1) a class-agnostic grouping task to group the pixels into segments.2) a zero-shot classification task on segments. The former sub-task does notinvolve category information and can be directly transferred to group pixelsfor unseen classes. The latter subtask performs at segment-level and provides anatural way to leverage large-scale vision-language models pre-trained withimage-text pairs (e.g. CLIP) for ZS3. Based on the decoupling formulation, wepropose a simple and effective zero-shot semantic segmentation model, calledZegFormer, which outperforms the previous methods on ZS3 standard benchmarks bylarge margins, e.g., 35 points on the PASCAL VOC and 3 points on the COCO-Stuffin terms of mIoU for unseen classes. Code will be released athttps://github.com/dingjiansw101/ZegFormer.