The resonant Rayleigh scattering dynamics of excitons in single GaAs quantum wells is investigated. The deviation of the measured intensity from the ideal ensemble average is analyzed as a function of the speckle ensemble size. The influence of the amplitude and correlation length of the exciton disorder potential is traced using a series of samples with varying inhomogeneous broadening and interface island sizes. The experimental data are compared with theoretical predictions using exciton states of spatially uncorrelated energies, as well as full calculations of the exciton polarization dynamics using a more realistic disorder potential including the formation of monolayer islands. Deviations from the dynamics of uncorrelated states are f...