Visual-Linguistic Reasoning with Deep Learning

Understanding single modality mediums including audio, visual, and language have achieved great success due to the high learning ability of neural network modules such as Convolution Neural Network (CNN), Long Term Short Term Memory (LSTM), Transformer and MLP-Mixer. However, a joint understanding of language and visual medium is still challenging as features in different modalities have a different distributions and lie in different manifolds.Thanks to the development of better visual and langauge backbone design, effective attention mechanism, and better multi-modality bilinear fusion approaches, the performance of Vision Linguistic Reasoning (VLR) systems, including, Visual Question Answering, Image Captioning, and Referring Expression have been improved significantly. However, the current design of VLR system is still challenging as most methods abandon spatial information in grid-level visual features. Moreover, only intra-modality relationship has been considered while intra-modality fusion has been omitted. Besides, multi-modality methods are only tested on visual-language reasoning system while its applicability can go beyond it.In this dissertation, I address the challenges presented before. In the first part, I will introduce a Question-Guided Hybrid Convolution (QGHC) network which can tackle efficient multi-modality fusion between gird features and language. QGHC can capture spatial information efficient during multi-modality fusion. In the second part, I will introduce the Dynamic Fusion of Intra- and Inter-Modality Attention Flow (DFAF) module, which performs bidirectional information exchange between language and object-level features utilizing the key-query attention mechanism. DFAF can jointly captures both intra- and inter-modality relationships. In the third part, we introduce a Multi-modality Latent Interaction (MLIN) module which can solve the problem of quadratic computation cost in DFAF using latent interaction. Last, we present Spatially Modulated Co-Attention (SMCA) mechamism, a high-quality object detection module motivated by a multi-modality interaction mechanism. SMCA demonstrate the general applicability of multi-modality system.Our proposed methods can be simple, compact, interpretable, and scalable compared with previous multi-modality fusion methods. Besides, we successfully transfer the techniques from multi-modality to general vision recognition tasks and show the broad applicability of our proposed approaches.卷積神經網絡，長短時間循環網絡，純注意力網絡和純多層感知混合網絡在音頻，圖像和語言的理解上面獲得了突飛猛進的發展。但是，如何將神經網絡技術應用到多模態融合依然存在著很多挑戰。原因是因為不同模態的特徵分佈不同，並且劉形也不一樣。推進神經網絡再多模態的研究可以讓我們離通用的人工智能更進一步。多模態融合方法包含視覺問答，圖像自然語言描述，指稱表帶等重要任務，在過去幾年獲得飛速發展。主要的原因在於更好的視覺和語言特徵，更有效的注意力和多模態雙線型融合機制。但是，現行的多模態融合方法依然有以下幾個問題。第一，多數多模態融合方法沒有充分利用網格特徵中存在的空間信息。第二，多模態融合系統只能夠捕捉跨模態的信息流動。模態內部的信息流動被忽略。第三，多模態方法僅僅在圖像語言推理任務上使用。這限制了多模態方法的應用範圍。為了推進神經網絡在多模態問題上面的應用，我們貢獻了一下的方法來解決上面提到的挑戰。第一，我們提出了問題引導的卷積神經網絡。問題引導的卷積神經網絡可以有效的將網格特徵跟語言特種高效的進行融合。問題引導的卷積可以充分利用圖像網格信息的空間信息。第二，我們提出了動態模態內和模態間注意力流來進行物體層面的特徵與語言的融合。我們的方法可以同時捕捉模態內和跨模態的信息流動。第三，為了解決動態模態內和模態間注意力流中平方計算量的問題，我們提出了多模態隱空間交互的方法來降低計算複雜度。最後，受到多模態研究啟發，我們提出了空間調製的相互注意力來進行物體檢測，相比於之前的方法DETR，空間調製的注意力可以取得更好的結果和10倍的收斂速度。空間調製注意力在物體檢測的成功說明了多模態交互的方法的廣泛的適用性。我們提出的方法相比於前人更加簡單，緊緻，容易解釋並且易於擴展。並且，我們也在通用視覺問題上面證明了多模態方法廣泛的適用性。GAO, Peng.Ph.D. Chinese University of Hong Kong 2021.Includes bibliographical references (leaves )Abstracts in English and Chinese.Title from PDF title page (viewed on ...