Dot-depth, monadic quantifier alternation, and first-order closure over grids and pictures

AbstractThis paper presents results from two different areas. The first area is monadic second-order logic (MSO) over finite structures, in particular over the so-called grids. These are structures whose elements can be arranged as a matrix and which have two binary relations corresponding to vertical and horizontal successors. For this logic, we study the expressive power of the alternation of existential and universal monadic second-order quantifiers, i.e., set quantifiers. In Matz et al. (Information and Computation, LICS’ 97, 1999, to appear) it had been shown that these alternations cannot be limited to a fixed number without loss of expressiveness. In this paper, we strengthen this result in several ways. Firstly, we show that there are MSO formulas that have a very restricted form of k+1 set quantifiers but are not equivalent to a formula with k quantifiers. Secondly, we show that if we fix the number of such alternations, the expressive power of formulas that start with a block of universal quantifiers differs from the power of those that start with an existential one—this was previously known only for coloured grids. Thirdly, we investigate how an additional prefix of first-order (i.e., element) quantifiers influences the expressive power of MSO formulas. The second area that this paper is concerned with is two-dimensional formal language theory. We study how the alternation of (first- and monadic second-order) quantifications, on the one hand, relates to the dot-depth measure of two-dimensional (i.e., picture) languages, on the other hand. That measure is the two-dimensional version of the classical notion of dot-depth for (one-dimensional) starfree word languages. We show that the hierarchy induced by this dot-depth cuts through the hierarchy given by monadic second-order quantifications. In particular, beyond each level of the monadic second-order alternation hierarchy, there is a starfree picture language