Submitters:

Nitzan Shapira – Yael Wiesel

Supervisor:

Mr. Michael Kolomenkin

Spring 2009

Project Goal:

The goal of the project is to create a system which receives a distances image from a depth camera (RGBD), and creates a virtual 3-D world which changes according to the changes in the real world.

The camera will take pictures of a plane, on which simple objects will be placed. The simple object will appear in the virtual world as complex objects, according to the desirable application.

The positions, sizes and proportions of the objects in the virtual world will be correlated to those of the objects in the real world, in order to create a realistic view of the real world.

The virtual world will react to changes in the plane, and the user will be able to have active affects in the virtual world.

Tools:

RGBD camera, created by 3DV Systems:

- The camera provides a distance image, as well as a standard color image.

- The camera is effective in the distance range: 0.7m – 1.5m.

- The camera has a capability of taking pictures in the dark.

Graphics Engine – OGRE:

- OGRE – Object-Oriented Graphics Rendering Engine

- Written in C++ - efficient.

- Object oriented – easy to use.

The Project Map:

The system has several parts:

- Calibration:
The calibration is performed once. The purpose of the calibration is to create compatibility between the distances of the objects in the real world to the number of pixels in the image received by the camera.

- Reading a frame from the camera:
In order to read frames from the camera in real-time, we used a program which was written by 3DV-Systems, with several changes we made in it.

- Frame processing by MATLAB:

During the run time of the system, MATALB processes each frame received from the camera. The processing includes: cleaning noises in the input image, background subtraction in order to identify objects, threshold operation and morphological operations on the difference image. All of these operations are performed in order to overcome many technical problems in the camera which make working with the camera a difficult issue.

- Objects identification:

After the frame processing, MATLAB identifies the objects which were placed on the plane and sends their statistics to OGRE.

- Placing the objects in the virtual world by OGRE:

The OGRE reads the objects’ statistics which were sent to it by MATLAB, and places corresponding 3-D objects in the virtual world.

- Interaction:

While running, the user can affect the application using the mouse and the keyboard. In some of the applications, a character can be moved on the surface using the keyboard.

Results:

In order to demonstrate the applicability of the system, we implemented several applications which use it.

Here are some images from these applications. More information can be found in the project book.

Links:

Final Presentation
Project Book
Source Code

Demo movies:

Demo 1 - "Walking on Hills"
Demo 2 - "Hunting"
Demo 3 - "Eating"
Demo 4 - "Sleeping"