Phases en compétition dans un système d'électrons fortement corrélés avec frustration

In this thesis we use the Variational Cluster Approximation (VCA) in the investigation of broken symmetry states of strongly correlated systems with frustration. Layered organic compounds, in which dimers of organic molecules form an anisotropic triangular lattice, are among materials that show this frustration. We discuss the two-dimensional one-band Hubbard model used for studying these compounds. Then we introduce VCA, which allows to study ordered phases by a variational principle based on the electron self-energy. We explain the computational methods that we used in conjunction with VCA. A comparison of the normal state and Néel antiferromagnetic ordered phase energies enables us to conclude that this order is dominant at large values of U, below some critical value of frustration (t'/t ). By observing the saturation of the order parameter, we argue that U [greater or approximately equal to] 8 is already in the strong coupling limit. d -wave superconductivity is discussed in relation with cluster and lattice point group symmetries. The two different pairings, d[subscript x[superscript 2]]-[subscript y[superscript 2]] and d[subscript xy] , are studied separately. A comparison of the energies of the antiferromagnetic and superconducting phases shows that while d -wave superconductivity dominates the antiferromagnetic phase, the d[subscript x[superscript 2]]-[subscript y[superscript 2]] order exists at intermediate U and d[subscript xy] is dominant at low values of U. We found no evidence of homogeneous coexistence of antiferromagnetic and d -wave superconducting phases. In addition, we investigate a spiral magnetic order on the isotropic triangular lattice, where no Néel antiferromagnetic order is found. By looking at the density of states, we see that the system is metallic at weak coupling. For U [greater or approximately equal to] 6 until a value in the range [8,12), we find an insulating phase, without long-range order, which we conjecture to be a spin liquid phase. This spiral order is found at stronger coupling