Painterly Rendering with Coherence for Augmented Reality

Figure 1: A sequence of painted AR frames. A real green cup, a real black cover book, a virtual teapot and a virtual bunny on a table. A seamless blending of the real and virtual worlds is key to increased immersion and improved user experiences for augmented reality (AR). Photorealistic and non-photorealistic rendering (NPR) are two ways to achieve this goal. Non-photorealistic rendering creates an abstract version of both the real and virtual world by stylization to make them indistinguishable. We present a painterly rendering algorithm for AR applications. This algorithm paints composed AR video frames with bump-mapping curly brushstrokes. Tensor fields are created for each frame to define the direction for the brushstrokes. We use tensor field pyramids to interpolate sparse tensor field values over the frame to avoid the numeric problems caused by global radial basis interpolation in existing algorithms. Due to the characteristics of AR applications we use only information from the current frame and previous frame to provide temporal coherence in two ways for the painted video. First, brushstroke anchors are warped from the previous frame to the current frame based on their neighbor feature points. Second, brushstroke appearances are reshaped by blending two parameterized brushstrokes to achieve better temporal coherence. The major difference between our algorithm and existing NPRwork in general graphics and AR/VR areas is that we use feature points across AR frames to maintain coherence in the rendering. The use of tensor field pyramids and extra properties of brushstrokes, such as cut-off angles, are also novel features that extend exiting NPR algorithms