Outlier Identification and Large-Scale Inference

在許多應用裡奇異辨識是經常出現的課題，且依然是熱門且未解決 的問題。從資料分析的角度來看，奇異辨識在現今的實際情境上主要 有兩個挑戰，其一是資料壓縮的問題，如何從巨量原始資料萃出高訊 息的資料，其二是如何利用有效統計推論從未知結構的大尺度資料中 發掘訊息。傳統的統計方法面對現代大尺度資料分析的問題有無法掩 飾的缺點，一般來說表現隨著尺度增加而衰退，使其在資料取樣不足 時往往不能達到令人滿意的表現。這篇論文使用了感測器網路當例子 闡釋了奇異辨識的一般性推論程序，這論文提出了系統性的架構用以 建立有效的奇異檢測, 提出的辨識方法以縮小估計式如Jame-Stein 估計 式為基礎，且在大尺度資料的情境下具有優勢，除此之外這本論文提 出了以充水性演算法為基礎的程序解最佳劃問題，用以求出漸進最佳 的縮小估計式，並可以用到更廣泛的資料分析應用。Outlier detection is a frequently encountered technology challenge for many diverse applications, and remains an open problem in general. There are two major difficulties of developing outlier detectors with collected data. One is the inevitable data reduction, the other is the effective inference when discovering information from unknown structured large-scale data. It is even more interesting and challenging with limited observation, since conventional data analysis requires many samples to achieve a satisfactory performance. In this thesis, a sensor network example is used to illustrate a general inference procedure for outlier identification. A systematic framework is proposed to develop effective and efficient outlier identifiers using shrinkage methodology, like James-Stein estimator, as the post-processor. This thesis show the superiority of our approach, particularly for the large-scale situations. This thesis further supply a water-filling type algorithm to obtain the asymptotic optimal method for a general class of shrinkage estimators, for wide applications of data analysis