A New Auto Document Category System by Rough Set Theory And N-gram Extraction

[[abstract]]現今資訊的發達，資料的流動十分迅速且繁多，想從資料找出重點很不容易，本文利用粗糙集，粗糙集是種處理不精確、不確定和不完全數據的數學方法，被廣汎地應用於資料探勘(Data Mining)的研究中，由於文本資料的繁雜和不確定性，能模糊集合所分析出的關鍵字，因此使用篩選出的關鍵字來判斷，用此方法應用至本文資料集，分析關鍵字並代入測試資料辨別類別。 本文總共做了三次實驗，首先共732筆文件抽取150筆當作測試檔，測試的資料庫內有九個類別共26833筆單字中，找出10229筆類別特有字，除了特定類別，部份F1值平均都有0 7以上。第二次從732隨機抽取一定數量並多次實驗，說明我們實驗樣本數量多的類別數據比較好且穩定。第三次則是類別內相同數量樣本文件數，數據都有九成左右，用來證明我們的數量影響辨識成功率的結論是正確的。最後根據我們三次實驗說明粗糙集理論是可以應用於分類，並有效提升文件分析處理速度。[[abstract]]In a modern society data is flowing quickly It's hard to extract keywords from the data In this paper we use rough set to deal with imprecise uncertain and incomplete data Rough sets are widely used in data mining research because rough sets are able to help us to analyze the keywords of text data Therefore we use Rough set to filter out the keywords There are three experiments executed in this research First from a total of 732 files to extract 150 files as a test file These test files contain nine categories of 26833 keywords and we use the Rough Set to find the 10 229 unique keywords for each category Except to specific categories most of the categories' analyzed results (F1 values) are above 70% In the second experiment we randomly selected a certain number test files from 732 files After several times of cross validation it indicates that the more of the number of samples has a better classification results The third experiment uses the same number of samples as the second experiment and their F1 values are about 90% Finally according to the three experiments it illustrates that the rough set theory can be applied to text classification and effectively enhance the text analysis and processing spee