Nonparametric estimation of random effects densities in linear mixedeffects model

Abstract. We consider a linear mixed-effects model where Yk,j = αk+βktj+εk,j is the observed value for individual k at time tj, k = 1,..., N, j = 1,..., J. The random effects αk, βk are independent identically distributed random variables with unknown densities fα and fβ and are independent of the noise. We develop nonparametric estimators of these two densities, which involve a cutoff parameter. We study their mean integrated square risk and propose cutoff-selection strategies, depending on the noise distribution assumptions. Lastly, in the particular case of fixed interval between times tj, we show that a completely data driven strategy can be implemented without any knowledge on the noise density. Intensive simulation experiments illustrate the method