使用影像與文字對齊技術摘要部落格文章;A Study on Summarizing Blog Articles by Using Image-Text Alignment

[[abstract]]現在部落格文章資訊越來越豐富，但是對於想快速地進行瀏覽或使用行動裝置瀏覽的使用者是相當不方便的，使用者常須不斷捲動網頁進行瀏覽。在本論文中，我們要進行部落格文章摘要來幫助這些使用者。我們觀察到以前的文章摘要技術幾乎都沒有參考影像資訊，當一篇文章有影像時，影像豐富文章內容，也提供重要的資訊。所以，我們將利用影像搭配文字資訊來做文章摘要，大大增加摘要的可看性，且大幅減少使用者的捲動行為。這裡我們利用近幾年發展相當快速的深度學習技術來實作影像與文字對齊技術的文章摘要系統。 首先，偵測出影像中許多可能是物件的區塊，每個物件區塊利用卷積神經網路萃取視覺特徵。文字部分先利用word2vec取文字特徵，並透過適當的轉換。之後再將視覺及文字兩種不同的特徵投影到相同的特徵空間，當文字和影像在相同特徵空間時，我們就可以直接計算兩者間的距離，透過這個距離我們可以知道影像與文章中的哪個句子語意相近，最後選出最適當的影像及文字配對形成文章摘要結果。 在實際的部落格實驗中，我們的方法勝過直接以image captioning結果做文章摘要，還有基於News2Image概念的文章摘要。我們也試著考慮文章與影像的空間特性加強文章摘要結果，並探討未來可再加強那些部分。 Rich content appears in blog articles currently. However, lengthy articles impede efficient access and often prevent users from reading articles from mobile devices. We thus would like to summarize blog articles to facilitate mobile browsing. Although images are often embedded in blog articles, previously document summarization works seldom consider image information. In this thesis, we would like to jointly consider image and text information, and summarize blog articles into image-sentence pairs so that readability can be enhanced. We will take advantage of deep learning technologies to achieve image-text alignment. Object regions in an image are first detected. From each object region we extract CNN features. On the other hand, text in sentences is transformed into vectors by the word2vec module. We then map image features and text features into a common space, where we can compute distance between sentences and images. Finally, semantics-related image-sentence pairs are determined to form blog article summaries. In the evaluation, we show that the result of our system is better than directly calculating similarity between sentences and image captions automatically generated by an image captioning engine. Our performance is also superior to a similar system, News2Image. We also try to consider spatial information of images to improve performance, and discuss future works