Robust and nonparametric methods for skewed data.

Statistics deals with gaining information from data. In practice, data often contain some randomness or uncertainty. Statistics handles such data using methods of probability theory. Most of these methods, rely on assumptions that are not always met in practice. In particular, it is often assumed that data follow a certain probability distribution. Even if the distributional assumptions are valid for the majority of the data, the presence of outliers can influence dramatically the outcome of the classical methods.An outlier is an observation that lies far away from the rest of the data. Outliers can occur by chance in any distribution, but they are often indicative of measurement error. Robust statistics is the art of minimizing the influence of outliers. The aim is to fit a model that is followed by the majority of the data.Chapter 1 deals with the detection of outliers in multivariate skewed distributions. To detect outliers in multivariate data, it is common practice to estimate the location and scatter of the data by means of robust estimators. All these estimators assume that the data are generated from an elliptical distribution, among which the multivariate gaussian is the most popular one. Consequently these outlier detection methods will not work appropriately when data are skewed. Our method is inspired by the Stahel-Donoho estimator. This estimator is based on the outlyingness of the data points, which is essentially obtained by projecting the observations on many univariate directions and computing a robust center and scale in each projection.In order to adjust for skewness, the Stahel-Donoho outlyingness is replaced by the adjusted outlyingness. The method is based on the adjusted boxplot for skewed data and essentially defines for univariate data a different scale on each side of the median. This scale is obtained by means of the medcouple. In the second step of our outlier detection method, we declare an observation as outlying when its adjusted outlyingness is 'too' large. In Chapter 2, the concept of 'adjusted outlyingness' is used for classification. If we can determine for an observation whether it is outlying with respect to a given population, it is also possible to say whether or not that observation is likely to belong to that population. This is essentially what is called a classification problem: given a random sample from a group of k populations, constructing a rule to classify a new observation into one of the k populations. Many of the existing classification methods rely on quite strict distributional assumptions such as multivariate normality, or at least elliptical symmetry. Proposals for skewed data are hardly available. In this chapter, we propose classification rules for skewed data, based on the adjusted outlyingness.The most important contribution of the first 2 chapters is the extension of existing outlier detection rules and classifiers to skewed distributions. In order to achieve this goal, it is of major importance to estimate the skewness as accurately as possible. The skewness estimator we used is the medcouple. One of the main questions addressed in Chapter 3, is whether it is possible to improve this skewness estimate, but also other estimators that are based on quantiles are studied. In order to determine whether one estimator is better than another, minimal mean squared error is often used as a criterion. In order to reduce the mean squared error, the estimators are based on a smoothed distribution function instead of the empirical distribution function. Smoothing requires the choice of a bandwidth. It is shown that an appropriate choice of this parameter results in an important reduction in mean squared error.In Chapter 1 and 2, classical methods were adapted for skewness. Another possible solution is to transform asymmetric data to symmetry and apply the classical methods on these transformed data. Chapter 4 describes how transformations can be done in a robust way. The most famous transformation is the Box-Cox transformation. However, this transformation has the drawback that it can only be applied to positive values. Yeo and Johnson proposed a more general transformation that can handle all real values.The estimation of the transformation parameter relies on maximum likelihood (ML) equations. It is generally known that ML-estimates are very sensitive for outliers.Therefore, a maximum trimmed likelihood approach is followed. Transformations to multivariate symmetry are also considered.status: publishe