We consider quantum diffusion in ultraslow-roll (USR) inflation. Using the Delta N formalism, we present the first stochastic calculation of the probability distribution P(R) of the curvature perturbation during USR. We capture the nonlinearity of the system, solving the coupled evolution of the coarse-grained background with random kicks from the short wavelength modes, simultaneously with the mode evolution around the stochastic background. This leads to a non-Markovian process from which we determine the highly non-Gaussian tail of P(R). Studying the production of primordial black holes in a viable model, we find that stochastic effects during USR increase their abundance by a factor of similar to 10(5) compared with the Gaussian approxi...