Methods for imputation of missing values in air quality data sets

環保署於1994年起為了要進行空氣品質趨勢探討，並了解各地的污染情形和各種污染的傳輸情形以及制訂合理改善控制對策，於是就依各地污染特性、地形及氣象條件等，將台灣地區劃分成七個空氣品質區，至今共設立了超過七十個空氣品質監測站。 因為有監測站全天候的監測各地空氣品質狀況，使得現在有環境空氣品質資料庫的歷史資料供大家研究及參考，可經由這龐大的資料庫中找尋出我們所需要的資訊。但監測站網所收集的空氣汙染物資料，只在傳輸的過程中就有約10%的資料因此遺失，若再加上其他因素的影響，導致更多資料的遺失。而資料庫中遺失值的存在，往往會影響到資料分析的品質，所以如何妥善處理這些遺失值是非常重要的關鍵。本研究分別利用距離平方反比與克利金兩種模式對空氣品質資料中的遺失值進行補值，並分析討論各補值後的效果，再經由蒙地卡羅(Monte Carlo)交叉驗證，探討及分析何種模式對於遺失值回復的準確性是比較好的。 經研究實驗模擬後發現，對於空氣品質資料庫中的遺失值進行補值時，克力金模式模擬補值的相對誤差比距離平方反比法的相對誤差小，且經蒙地卡羅法交叉驗證，綜合五個汙染物PM10、CO、O3、SO2、NOx結果來說，克力金法的平均相對誤差為25%，距離平方反比法的相對誤差為19%，顯示克力金模式比距離平方反比法是更適用的插值模式。In order to discuss the problem of the air quality, to understand the pollution and the different pollution transmission in each place, and to design strategies to control and improve the contamination, Environmental Protection Administration in Taiwan in 1994 divided Taiwan into 7 air quality districts based on their different pollution features, the terrain, and the climate in each district. They have already set up over 70 air quality monitoring stations in Taiwan. Because of the air quality monitoring stations which monitored air quality condition of each place, we are able to have the historical information of air quality of everywhere for everybody to research and find out the useful information from this huge database. But around 10% of data are missing only during the transmission process. However, the missing values of the database will affect the data analysis, hence, it is very important to resolve the missing value problem. My research uses Inverse Square Distance Weighting method and Kriging method to impute the missing values, and discusses, analyzes, and compares the result of using the 2 different methods. Monte Carlo method is then used to test and verify which one is the better method to yield the accurate values to replace the missing values. After my research, the absolute error of Inverse Square Distance Weighting is smaller than it of Monte Carlo method for imputation of air quality data. After verifying, the absolute error of Inverse Square Distance Weighting and Kriging method is respectively 25% and 19%. It shows that Kriging method is better imputation method than Inverse Square Distance Weighting.誌謝......................................................i摘要....................................................iii 目錄....................................................vii 表目錄...................................................xi 圖目錄.................................................xiii 第一章 前言...............................................1 1.1 研究背景與動機....................................1 1.2 研究重要性........................................2 1.3 研究目的..........................................2 第二章 文獻回顧...........................................3 2.1 遺失值的介紹......................................3 2.1.1遺失值的定義與成因............................3 2.1.2 遺失值的種類與型態...........................4 2.1.3 遺失值的處理方式.............................5 2.2 國內外相關文獻....................................7 2.2.1空間推估的相關文獻............................7 2.2.2 可靠性相關文獻...............................9 第三章 研究方法與流程....................................11 3.1 資料來源與說明...................................12 3.2 資料前處理.......................................15 3.3模式理論導入......................................16 3.3.1 距離平方反比法..............................16 3.3.2克力金法.....................................17 第四章 研究結果與討論....................................21 4.1預測的正確性......................................21 4.2模式模擬結果......................................22 4.2.1距離平方反比法模擬結果.......................22 4.2.2 克力金法模擬結果............................28 4.2.3模擬結果分析與討論...........................34 4.2.4 IDW和Kriging的比較..........................38 4.3模式模擬方法分析..................................53 4.3.1距離平方反比模式方法分析.....................53 4.3.2 克力金模式方法分析..........................60 4.4交叉驗證..........................................77 4.4.1蒙地卡羅交叉驗證法...........................77 4.4.2交叉驗證結果.................................78 第五章 結論與建議........................................81 5.1結論..............................................81 5.2建議..............................................82 參考文獻.................................................83 附錄一...................................................87 附錄二...................................................8