A Plane-based Approach to Mondrian Stereo Matching

Stereo vision is the problem of estimating a 3D depth map—encoded as pixel displacements, or disparities—of a scene from two images taken from adjacent viewpoints. Disparities can be computed by finding for each pixel in... [ view full abstract ]

Stereo vision is the problem of estimating a 3D depth map—encoded as pixel displacements, or disparities—of a scene from two images taken from adjacent viewpoints. Disparities can be computed by finding for each pixel in the left image the best color match in the right image. However, this strategy does not work well in completely untextured regions, e.g. blank walls, where a pixel in the left image can easily match to any number of pixels in the right.

The goal of this project was to devise a stereo algorithm that can handle even the pathological case of synthetic scenes consisting solely of solid-colored regions, resembling the abstract paintings by Dutch artist Piet Mondrian. Whereas existing algorithms rely on matching image texture, my method matches edges of color-based image segments and uses their 3D locations to fit 3D disparity planes to the segments. Throughout the summer we generated increasingly complex test image pairs, designed new implementations that solves these, and repeated the process with new, more challenging images. My algorithm currently solves all test cases we have developed, where existing state-of-the-art algorithms consistently fail. This holds promise for using similar concepts and strategies in robust algorithms for real-world images.