Comparison of the properties of two fossil groups of galaxies with the normal group NGC 6034 based on multiband imaging and optical spectroscopy

International audienceContext. Fossil groups are dominated by a bright galaxy, and their luminosity functions show an absence within half the virial radius of galaxies brighter than the central galaxy magnitude +2. They are nevertheless massive with an extended X-ray halo. The formation and evolution of these structures is still widely debated. Aims. To better understand the origin of these structures, it is crucial to study their faint galaxy population, as well as their large-scale environment, to determine in particular whether they are isolated or not. Methods. We collected multiband imaging and spectroscopy for two fossil groups (RX J1119.7+2126 and 1RXS J235814.4+150524) and one normal group (associated with NGC 6034). We computed photometric redshifts in the central zones of each group, combining previous data with the SDSS five-band data. For each group we investigated the red sequence (RS) of the color-magnitude relation and computed the luminosity functions, stellar population ages and distributions of the group members. Spectroscopy allowed us to investigate the large-scale surroundings of these groups and the substructure levels in 1RXS J235814.4+150524 and NGC 6034. Results. The large-scale environment of 1RXS J235814.4+150524 is poor, though its galaxy density map shows a clear signature of the surrounding cosmic web. RX J1119.7+2126 appears to be very isolated, while the cosmic environment of NGC 6034 is very rich. At the group scale, 1RXS J235814.4+150524 shows no substructure. Galaxies with recent stellar populations seem preferentially located in the group outskirts. A red sequence is discernable for all three groups in a color-magnitude diagram. The luminosity functions based on photometric redshift selection and on statistical background subtraction have comparable shapes, and agree with the few points obtained from spectroscopic redshifts. These luminosity functions show the expected dip between first and second brightest galaxies for the fossil groups only. Their shape is also regular and relatively flat at faint magnitudes down to the completeness level for RX J1119.7+2126 and NGC 6034, while there is a clear lack of faint galaxies for 1RXS J235814.4+150524. The faint parts of the luminosity functions appear dominantly populated by late-type galaxies. Conclusions. RX J1119.7+2126 is definitely classified as a fossil group; 1RXS J235814.4+150524 also has properties very close to those of a fossil group, while we confirm that NGC 6034 is a normal group