A Single Camera Gaze-Replacement Method

by Tamir Raz and Gabi Guetta

Supervised by Dmitry Rudoy

Abstract

The camera in video conference systems is typically positioned above, or below, the screen, causing the gaze of the users to appear misplaced. We propose an effective solution to this problem that is based on replacing the eyes of the user. This replacement, when done accurately, is enough to achieve a natural looking video. At an initialization stage the user is asked to look straight at the camera. We store these frames, then track the eyes accurately in the video sequence and replace the eyes, taking care of illumination and ghosting artifacts. We have tested the system on a large number of videos demonstrating the effectiveness of the proposed solution.

Tools

This project was implemented in a Matlab environment, based on one main article : Lior Wolf et al.

General Scheme

         Figure 1 - General scheme of the algorithm.

Algorithm

Stage 1 – Building an offline database

The direct gaze models are learned based on previous annotated examples. These examples are stored in an offline database that is used to bootstrap the runtime system. It consists of a separate set of 14 individuals for whom videos were captured and manually annotated in accordance with the six parameter model.

     Figure 2 – Eyes from the offline database

Stage 2 – Building an online database

Given images of a new person looking directly into the camera, the closest eye in the offline database is retrieved and then serves as an initial guess as to the eye parameters in the new image. These parameters are then adapted to provide a more accurate estimation.

        Figure 3 – Eyes from the online database

Stage 3 – Runtime eye Localization

At runtime, the parameters of the eyes in the video are estimated by matching the image to the eyes of the person when looking straight. The illumination is corrected, and a replacement is performed.

                                                                  Figure 4 – Example for the Enlightment Adjusting+Feathering procedure

Results

DOWNLOADS

• Project Report [MS Word ]
• Presentation[MS Powerpoint ]
• Source Code [Zip]