Euclid preparation. XVIII. The NISP photometric system

Euclid will be the first space mission to survey most of the extragalactic sky in the 0.95-2.02 $\mu$m range, to a 5$\sigma$ point-source median depth of 24.4 AB mag. This unique photometric data set will find wide use beyond Euclid's core science. In this paper, we present accurate computations of the Euclid Y_E, J_E and H_E passbands used by the Near-Infrared Spectrometer and Photometer (NISP), and the associated photometric system. We pay particular attention to passband variations in the field of view, accounting among others for spatially variable filter transmission, and variations of the angle of incidence on the filter substrate using optical ray tracing. The response curves' cut-on and cut-off wavelengths - and their variation in the field of view - are determined with 0.8 nm accuracy, essential for the photometric redshift accuracy required by Euclid. After computing the photometric zeropoints in the AB mag system, we present linear transformations from and to common ground-based near-infrared photometric systems, for normal stars, red and brown dwarfs, and galaxies separately. A Python tool to compute accurate magnitudes for arbitrary passbands and spectral energy distributions is provided. We discuss various factors from space weathering to material outgassing that may slowly alter Euclid's spectral response. At the absolute flux scale, the Euclid in-flight calibration program connects the NISP photometric system to Hubble Space Telescope spectrophotometric white dwarf standards; at the relative flux scale, the chromatic evolution of the response is tracked at the milli-mag level. In this way, we establish an accurate photometric system that is fully controlled throughout Euclid's lifetime