Label Relation Based Scene Classification Using CNNs and LSTM

在傳統的場景辨識方法中，通常假設每一個標籤是互斥的，但是這常常是不合理的，因為在場景的標籤中，可能會有一些關係，例如：雪山的場景同時屬於山跟雪兩個標籤，所以這是一個多標籤的場景辨識。其中，我們整理了兩個最主要的關係，階層式關係與互斥關係。希望透過這兩個關係來讓辨識結果更加的合理。 我們提出兩個方法，第一個方法是基於階層式的卷積類神經網路與關係圖結構，相對於傳統的假設標籤互斥的方法，我們假設圖的路徑是互斥的。但由於這個方法是需要對資料庫做預處理，同時需要人工建立關係圖。因此我們提出另外一個基於長短期記憶單元的方法，由於我們認為語言中的文法很像是標籤關係，因此我們透過長短期記憶單元的結構，來訓練出語言模型，並產生關於場景的敘述，這個敘述就是辨識的結果。從最後的模擬結果我們可以發現我們提出的兩個方法都比過往的多類別場景辨識結果要好，另外，基於長短期記憶單元的方法又比階層式卷積神經網路的方法好。In traditional scene classification, they assume the labels are mutually exclusive. But there are some relations between the labels. For example, the snow mountain scene must belong to both mountain and snow labels. Therefore, the results of the traditional label relations are not reasonable. We want to predict a more reasonable result based on the label relations. We conclude two relations, which are hierarchy relation and exclusive relation. We proposed two algorithms, the first algorithm is based on the hierarchy CNN and the label relation graph structure. We assume the paths in the graph are mutually exclusive instead of assuming the labels are mutually exclusive. But this algorithms need pre-processing of the dataset and we need to build the label relation graph in manual. Therefore, we proposed another algorithm which is based on the long short-term memory. The idea is the grammar between the words in the sentence is like the label relations between the labels. This is very like the image captioning work. Therefore, we train a language model to model the label relations and use the long short-term memory structure to produce the description of the image. The description of the image is our predict result. The simulation result suggests that the algorithms we proposed are better than other multi-label scene classification methods. In addition, the algorithm based on the long short-term memory is better than the algorithm based on the hierarchy convolutional neural network