INVARIANT NEUTRAL SUBSPACES FOR HAMILTONIAN MATRICES ∗

Abstract. Hamiltonian matrices with respect to a nondegenerate skewsymmetric or skewhermitian indefinite inner product in finite dimensional real, complex, or quaternion vector spaces are studied. Subspaces that are simultaneously invariant for the matrices and neutral in the indefinite inner product are of special interest. The dimension of maximal (by inclusion) such subspaces is identified in terms of the canonical forms and sign characteristics. Criteria for uniqueness of maximal invariant neutral subspaces are given. The important special case of invariant Lagrangian subspaces is treated separately. Comparisons are made between real, complex, and quaternion contexts; for example, for complex Hamiltonian matrices with respect to a nondegenerate skewhermitian inner product in a finite dimensional complex vector space, the (complex) dimension of (complex) maximal invariant neutral subspaces is compared to the (quaternion) dimension of (quaternion) maximal invariant neutral subspaces, and necessary and sufficient conditions are given for the two dimensions to coincide (this is not always the case)