We apply the formalism developed earlier for studying transverse momentum dependent parton distribution functions (TMDs) at small Bjorken $x$ to construct the small-$x$ asymptotics of the quark Sivers function. First, we explicitly construct the complete fundamental "polarized Wilson line" operator to sub-sub-eikonal order: this object can be used to study a variety of quark TMDs at small-$x$. We then express the quark Sivers function in terms of dipole scattering amplitudes containing various components of the "polarized Wilson line" and show that the dominant (eikonal) term which contributes to the quark Sivers function at small $x$ is the spin-dependent odderon, confirming the recent results of Dong, Zheng and Zhou. Our conclusion is als...