Time delays of electrons emitted from an isotropic initial state with the absorption of a single photon and leaving behind an isotropic ion are angle independent. Using an interferometric method involving XUV attosecond pulse trains and an IR-probe field in combination with a detection scheme, which allows for full three-dimensional momentum resolution, we show that measured time delays between electrons liberated from the 1s(2) spherically symmetric ground state of helium depend on the emission direction of the electrons relative to the common linear polarization axis of the ionizing XUV light and the IR-probing field. Such time delay anisotropy, for which we measure values as large as 60 as, is caused by the interplay between final quantu...