The polynomial conjecture for restrictions of some nilpotent Lie groups representations

International audienceLet $G$ be a connected and simply connected nilpotent Lie group, $K$ an analytic subgroup of $G$ and $\pi$ an irreducible unitary representation of $G$ whose coadjoint orbit of $G$ is denoted by $\Omega(\pi)$. Let $\mathcal U(\mathfrak g)$ be the enveloping algebra of ${\mathfrak g}_{\mathbb C}$, $\mathfrak g$ designating the Lie algebra of $G$. We consider the algebra $\left(\mathcal U(\mathfrak g)/\ker \pi\right)^K$ of the $K$-invariant elements of $\mathcal U(\mathfrak g)/\ker \pi$. It turns out that this algebra is commutative if and only if the restriction $\pi|_K$ of $\pi$ to $K$ has finite multiplicities (cf.\,A.\,Baklouti and H.\,Fujiwara, {\em Commutativit\'{e} des op\'{e}rateurs diff\'{e}rentiels sur l'espace des repr\'{e}sentations restreintes d'un groupe de Lie nilpotent}, J.\,Math.\,Pures\,Appl.\,83 (2004) 137--161). In this article we suppose this eventuality and we study the polynomial conjecture asserting that our algebra is isomorphic to the algebra $\mathbb C[\Omega(\pi)]^K$ of the $K$-invariant polynomial functions on $\Omega(\pi)$. We give a proof of the conjecture in the case where $\Omega(\pi)$ admits a normal polarization of $G$ and in the case where $K$ is abelian. This problem was partially tackled previously by A.\,Baklouti, H.\,Fujiwara, J.\,Ludwig, {\em Analysis of restrictions of unitary representations of a nilpotent Lie group}, Bull. Sci. Math. 129 (2005) 187--209