Non-Standard problems in inference for additive and linear mixed models.

Linear mixed models are a powerful inferential tool in modern statistics and have a wide range of applications. Recent advances utilize the connection between penalized spline smoothing and mixed models for efficient implementation of nonparametric and semiparametric regression techniques. These become increasingly important to adequately model the relationship between response variables and covariates. However, despite their common use, some open questions regarding the inference in mixed models still remain. This dissertation is aimed at improving the methodology for inference on random effects. An important special case is testing for polynomial regression against a general smooth alternative modeled by mixed model penalized splines. Our motivating application is the assessment of non-linearity for air pollution dose-response functions in the epidemiological Airgene study. Testing for a zero random effects variance is a non-standard testing problem. First, the tested parameter is on the boundary of the parameter space under the null hypothesis. Second, in linear mixed models observations are generally not independent. While in longitudinal linear mixed models there are at least independent subjects or units, such a subdivision of the data is not possible for mixed model penalized spline smoothing. We first investigate the asymptotic distribution of the restricted likelihood ratio test statistic when testing for polynomial regression using mixed model penalized splines. We show that asymptotic results on boundary testing for independent observations do not hold here. This is due to the asymptotic non-normality of the score statistic. Fundamentally, this is caused by the dependence of observations induced by penalized splines. We find that this dependence cannot be ignored, as it is inherently necessary for the attainment of smooth curves. Different approaches to this testing problem are therefore necessary. Subsequently, we provide finite sample alternatives for testing for zero random effect variances in linear mixed models. The class of models we consider is more general than has previously been covered, including models with moderate numbers of clusters, unbalanced designs, or nonparametric smoothing. We also allow more than one random effect in the model. We propose two approximations to the finite sample null distribution of the restricted likelihood ratio test statistic. Extensive simulations show that both outperform the chi-square mixture approximation and parametric bootstrap currently used, as well as several F-type tests. Finally, we discuss model selection for mixed model penalized splines using the Akaike Information Criterion. The criterion based on the marginal likelihood is found not to be asymptotically unbiased for the expected relative Kullback-Leibler distance. In fact, it is biased towards the simpler model. An alternative is provided using our results on restricted likelihood ratio testing