Coupling of FEM and BEM in shape optimization

In the present paper we consider the numerical solution of shape optimization problems which arise from shape functionals of integral type over a compact region of the unknown domain, especially $L^2$-tracking type functionals. The underlying state equation is assumed to satisfy a Poisson equation with Dirichlet boundary conditions. We proof that the shape Hessian is not strictly $H^1/2$-coercive at the optimal domain which implies ill-posedness of the optimization problem under consideration. Since the adjoint state depends directly on the state, we propose a coupling of finite element methods (FEM) and boundary element methods (BEM) to realize an efficient first order shape optimization algorithm. FEM is applied in the compact region while the rest is treated by BEM. The coupling of FEM and BEM essentially retains all the structural and computational advantages of treating the free boundary by boundary integral equations