We discuss some geometric problems related to the definitions of quasilocal mass proposed by Brown and York (Contemporary mathematics, vol 132, American Mathematical Society, Providence, pp 129-142, 1992; Phys Rev D (3) 47(4): 1407-1419, 1993) and Liu and Yau (Phys Rev Lett 90(23): 231102, 2003; J Am Math Soc 19(1): 181-204, 2006). Our discussion consists of three parts. In the first part, we propose a new variational problem on compact manifolds with boundary, which is motivated by the study of Brown-York mass. We prove that critical points of this variation problem are exactly static metrics. In the second part, we derive a derivative formula for the Brown-York mass of a smooth family of closed two dimensional surfaces evolving in an ambi...