We propose a general class of label configuration pri-ors for continuous multi-label optimization problems. In contrast to MRF-based approaches, the proposed frame-work unifies label configuration energies such as minimum description length priors, co-occurrence priors and hier-archical label cost priors. Moreover, it does not require any preprocessing in terms of super-pixel estimation. All problems are solved using efficient primal-dual algorithms which scale better with the number of labels than the alpha-expansion method commonly used in the MRF setting. Ex-perimental results confirm that label configuration priors lead to drastic improvements in segmentation. In particular, the hierarchical prior allows to jointly compute a semantic se...