In this work the Casimir-Polder interaction energy between a rubidium atom and a disordered graphene sheet is investigated beyond the Dirac cone approximation by means of accurate real-space tight-binding calculations. As a model of defected graphene, we consider a tight-binding model of π electrons on a honeycomb lattice with a small concentration of vacancies. The optical response of the graphene sheet is evaluated with full spectral resolution by means of exact Chebyshev polynomial expansions of the Kubo formula in large lattices in excess of 10 million atoms. At low temperatures, the optical response of defected graphene is found to display two qualitatively distinct behaviors with a clear transition around finite (nonzero) Fermi energy...