We study the deformations of twisted harmonic maps f with respect to a representation. After constructing a continuous "universal" twisted harmonic map, we give a construction of every first order deformation of f in terms of Hodge theory; we apply this result to the moduli space of reductive representations of a Kähler group, to show that the critical points of the energy functional E coincide with the monodromy representations of polarized complex variations of Hodge structure. We then proceed to second order deformations, where obstructions arise; we investigate the existence of such deformations, and give a method for constructing them, as well. Applying this to the energy functional as above, we prove (for every finitely presented grou...