Inference for non-random samples

Observational data are often analysed as if they had resulted from a controlled study, and yet the tacit assumption of randomness can be crucial for the validity of inference. We take some simple statistical models and supplement them by adding a parameter theta which reflects the degree of non-randomness in the sample. For a randomized study theta is known to be 0. We examine the profile log-likelihood for theta and the sensitivity of inference to small nonzero values of theta. Particular models cover the analysis of survey data with item nonresponse, the paired comparison t-test and two group comparisons using observational data with covariates. Some practical examples are discussed. Allowing for sampling bias increases the uncertainty of estimation and weakens the significance of treatment effects, sometimes substantially so