Observations in quantum weak measurements are determined by complex numbers called weak values. We present a geometrical interpretation of the argument of weak values of general Hermitian observables in $N$-dimensional quantum systems in terms of geometric phases. We formulate an arbitrary weak value in function of three real vectors on the unit sphere in $N^2-1$ dimensions, $S^{N^2-2}$. These vectors are linked to the initial and final states, and to the weakly measured observable, respectively. We express pure states in the complex projective space of $N-1$ dimensions, $\mathbb{C}\textrm{P}^{N-1}$, which has a non-trivial representation as a $2N-2$ dimensional submanifold of $S^{N^2-2}$ (a generalization of the Bloch sphere for qudits). T...