We perform ultrafast pump-probe spectroscopy of plasmonic structures beyond the perturbative excitation regime. As a prototypical system we choose gold nanorods dispersed in water, which display both longitudinal and transverse plasmon resonances. A broadband investigation reveals a complex scenario for the saturation of the transient optical response as a function of the pump fluence, with strong dependence on the probe wavelength. In particular, we observed stronger saturation effects for the high energy transverse plasmonic resonance as compared to the low energy longitudinal one. This behavior is well captured by a three-temperature model and is understood in terms of the nonlinearity of the Fermi-smearing mechanism, presiding over the ...