Context-Aware Short Text Segmentation based on Word Co-occurrence Model

文本探勘(Text Mining)於資訊爆炸的時代扮演著極為重要的角色與應用。進行文本探勘前，其最重要的前置步驟為斷詞(Sentence Segmentation)。斷詞結果的好壞明顯地影響後續的文本分析效果。斷詞問題於中文文本分析中更為明顯與重要。現有中文斷詞方法於正式文章，如新聞文章，已有良好的結果。然而針對短文且非正式的語句，如社群網路上之使用者留言，之斷詞效果仍有待改善。本論文中我們提出一個考量語意情境的斷詞方法；主要想法為透過維基百科的文章，建立一個共現詞詞典，在斷詞時透過共現詞詞典考慮短文前後文的共現詞關係來提升斷詞的準確度。我們透過實驗驗証，使用相同字典的情況下，考慮上下字詞的斷詞方式正確率較現有斷詞方法提高了7.84%。研究所開發之短文斷詞系統也公開原始碼置放於https://github.com/UDICatNCHU/CAS，供後續相關研究者進行研究使用。Text mining plays an extremely important role and application in the era of information explosion. Before any text mining techniques, the most important prerequisite is word segmentation. The quality of the word segmentation result significantly affects the performance of the follow-up text mining applications. The problem of word segmentation is more obvious and important in Chinese text analysis. The existing word segmentation method has had good results in official articles such as news articles. However, the performance of the existing word segmentation on short sentences and informal sentences still needs improvement. In this thesis, we propose a word segmentation method that takes into account the semantic context. The main idea is to create a co-occurrence dictionary by using Wikipedia articles, and use the co-occurrence dictionary to improve the performance of word segmentation. Through experimental evaluation, we verify that the accuracy of our proposal is 7.84\% higher than that of the existing methods.摘要......................................................................................................... i Abstract................................................................................................... ii 目錄......................................................................................... iii 表目次...................................................................................... v 圖目次...................................................................................... vi 第1 章緒論........................................................................................... 1 1.1 研究動機.................................................................................. 1 1.2 研究目的.................................................................................. 2 第2 章文獻探討.................................................................................... 4 第3 章研究方法.................................................................................... 5 3.1 Jieba 斷詞................................................................................ 5 3.2 Keyword Correlation Model: KCM.......................................... 9 3.2.1 Algorithm .................................................................. 12 3.3 考量上下文字詞共現關系的短文斷詞流程(context aware segmentation:CAS) .................................................................. 14 3.3.1 Algorithm .................................................................. 16 3.4 計算合理的詞頻....................................................................... 25 3.5 CAS 斷詞效率改善.................................................................. 27 第4 章實驗........................................................................................... 29 4.1 實驗比較.................................................................................. 29 4.2 實驗結論.................................................................................. 33 第5 章結論........................................................................................... 34 參考文獻.................................................................................. 35 程式使用說明........................................................................... 3