Yoav Y. Schechner: Research


Virtual Periscope

Distortions that are random in space, time and viewpoint are created when viewing objects through a wavy water-air interface. Such distortions are also created in turbulence. In both cases, the distortion is caused by dynamic refraction. We study how this situation can be handled. This is important to some predators. It is also beneficial to submariners, to avoid the salient use of a physical periscope. We have just devised a passive system (STELLA MARIS) to counter distortions caused by water waves, in a single-viewpoint and instant. This single-frame approach is deterministic: it can be stand-alone, or be an enabler for stochastic methods.

A stocahstic method we have developed localizes objects in three dimensions (3D) despite this random distortion. We take a multiview approach to this task. Refracted distortion statistics induce a probabilistic relation between any pixel location and a line of sight in space. Measurements of an object's random projection from multiple views and times lead to a likelihood function of the object's 3D location. The likelihood leads to estimates of the 3D location and its uncertainty. Furthermore, multiview images acquired simultaneously in a wide stereo baseline have uncorrelated distortions. This helps reduce the acquisition time needed for localization.

Moerover, in many cases, the moving objects are of interest, as they reveal animal, human, or machine activity. We show that moving objects can be detected very simply, with low false-positive rates, even when the distortions are very strong and dominate the object motion. While the object and distortion motions are random and unknown, they are mutually independent. This is expressed by a simple motion feature which enables discrimination of moving object points versus the background.


    1. M. Alterman, Y. Swirski and Y. Y. Schechner, "STELLA MARIS: Stellar Marine Refractive Imaging Sensor, Proc. IEEE ICCP - International Conference on Computational Photography (2014)
    2. Y. Y. Schechner, " A view through the waves, Marine Technology Society Journal 47, 148-150 (2013)
    3. M. Alterman, Y. Y. Schechner and Y. Swirski, "Triangulation in random refractive distortions," Proc. IEEE ICCP - International Conference on Computational Photography (2013).
    4. M. Alterman, Y. Y. Schechner, P. Perona and J. Shamir, "Detecting motion through dynamic refraction," IEEE Transactions on Pattern Analysis and Machine Intelligence 35, No. 1, pp. 245-251 (2013).


    1. STELLA MARIS: Stellar Marine Refractive Imaging Sensor. A narrated presentation in YouTube.
    2. Triangulation in Random Refractive Distortions. A narrated presentation in YouTube.
    3. Vision through random refractive distortions (89 Mb, PowerPoint).


      The following links contain raw material and results of our experiments:
      1. Monocular Jacuzzi experiment - Walking (204 Mb, Zip).
      2. Monocular Jacuzzi experiment - Hands waving (206 Mb, Zip).
      3. Stereo lab experiment (42.1 Mb, Zip).
      4. Stereo pool experiment - brooms (52.4 Mb, Zip).
      5. Stereo pool experiment - potato head (92.6 Mb, Zip).
      6. Related Research

        1. Stereo from Flicker
        2. Passive Turbulence Tomography
        3. Clear Underwater Vision in Natural Illumination
        4. Active Ilumination De-scattering

Refraction at a wavy water surface.
Sample stereoscopic images, looking upward through a wavy water surface. Stochastic triangulation.
Support of the airborne position likelihood.
Underwater stereo setup.
42.7 MB WMV
25.0 MB WMV
Motion Detection: [Top] Original sequence; [Bottom] Results overlaid


24.7 MB WMV
Triangulation: Original sequences, triangulation results with uncertainties.


Motion detection: the fixed theoretical classifier contour in feature space, and test scatter plots.