Eye-Contact for Video Conferencing Using Image Synthesis Based on Multi-Cameras

隨著網路技術的發達和普及，視訊會議以及即時聊天室等網路應用也越來越為廣泛，但是在使用視訊時的眼神交會問題仍然存在著。因為在使用視訊時如果沒有眼神的交會，使用者會覺得對方好像一直在看著其他的地方，產生一種錯誤的感覺。於是我們提出了一個系統，利用分層的影像處理來解決缺乏眼神交會的問題。藉由使用兩台已經事先校正過的相機，我們可以快速的生成出有眼神交會的影像。在一些前人提出來的解決方法中，有的會花很多的時間在運算方面，也有的會利用硬體的解決方式或加速來生成出有眼神交會的影像。在我們的解決方式中，我們是使用軟體的運算方式來解決問題，同時可以以接近即時的速度生成出我們所要的眼神交會影像出來。我們利用去背的方式，來將前景和背景分離，這樣我們可以更加專注在前景的部份來做處理。接著再利用Adaboost以及mean shift來偵測以及追蹤使用者的臉的所在位置。有了前景以及使用者臉的位置的資訊之後，我們可以減低運算所需要花的時間，同時增加我們在找尋對應點時的正確率。最後，我們將三層分開的影像 – 背景，合成出的前景部份，以及合成出的臉部 – 全部合在一起，就可以產生出有眼神交會的虛擬影像。In recent years, network is becoming more and more popular, and video conferencing and real time chat rooms are very common. However, eye contact problem still remains in our daily life. Without eye contact, people may feel very weird and not being noticed. In our work, we propose a system with layer image processing. Using two stereo cameras, which are pre-calibrated, we can synthesize quickly and get images with eye contact. In some previous works, eye contact can be generated with time consuming works or with some hardware support. In our work this can be done by software implement and near real time images can be synthesized. We use background segmentation to get the foreground part, which can help us concentrate on our ROI. Then we use Adaboost and mean shift to help us locate where the user’s face is. With the foreground information and the face location, we can improve our calculation time and get a better result. Finally, 3 layers of images – background, virtual foreground, and virtual face – will be put together to complete our synthesized image with eye contact.Contents 1 Introduction 1 1.1 Motivation 1 1.2 Problem definition 1 1.3 Major difficulties 2 2 Related work 4 3 System overview 7 3.1 System architecture 8 3.2 Camera calibration 10 3.3 Background segmentation 11 3.4 Image rectification 12 3.5 Face detection and tracking 15 3.5.1 Face detection 15 3.5.2 Face tracking 16 3.6 Virtual view synthesis 17 4 Virtual view synthesis 19 4.1 Image synthesis for the foreground layer 20 4.2 Image synthesis for the face layer 21 4.3 Image synthesis for the virtual view 25 5 Experimental results 26 6 Conclusion and future work 31 7 Bibliography 3