This paper aims at characterizing and improving the metrological performances of current and voltage instrument transformers (CTs and VTs) in harmonic measurements in the power system. A theoretical analysis is carried out to demonstrate that, due to the iron core nonlinearity, CT and VT output signal is distorted even when the input signal is a pure sine wave. Starting from this analysis, a new method for CT and VT characterization and compensation is proposed. In a first step, they are characterized in sinusoidal conditions and the harmonic phasors of the distorted output are measured; in the second step, these phasors are used to compensate the harmonic phasors measured in normal operating conditions, which are typically distorted. The p...