3D Reconstruction and Segmentation

As we navigate the world, for example when driving a car from our home to the work place, we constantly perceive the environment around us and recognise objects within it. Such capabilities help us in our everyday lives and allow us free and accurate movement even in unfamiliar places. Building a system that can automatically perform real-time semantic segmentation and 3D reconstruction is a crucial prerequisite for a variety of applications, including robot navigation, semantic mapping or assistive technology.
We propose an end-to-end system that can process the data incrementally and perform real-time dense stereo reconstruction and semantic segmentation of unbounded outdoor environments. The system outputs a per-voxel probability distribution instead of a single label (soft predictions are desirable in robotics, as the vision output is usually fed as input into other subsystems). Our system is also able to handle moving objects more effectively than prior approaches by incorporating knowledge of object classes into the reconstruction process. In order to achieve fast test times, we extensively use the computational power of modern GPUs.

Publications

Incremental Dense Semantic Stereo Fusion for Large-Scale Semantic Scene Reconstruction
Vineet V.*, Miksik O.*, Lidegaard M., Nießner M., Golodetz S., Prisacariu V.A., Kähler O., Murray D.W., Izadi S., Perez P. and Torr P.H.S.
In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) 2015, Seattle, USA
* Joint first authors
PDF | Show BibTex | Show Details


The Semantic Paintbrush: Interactive 3D Mapping and Recognition in Large Outdoor Spaces
Miksik O.*, Vineet V.*, Lidegaard M., Prasaath R., Nießner M., Golodetz S., Hicks S.L., Perez P., Izadi S. and Torr P.H.S.
In Proceedings of the 33nd annual ACM conference on Human factors in computing systems (CHI) 2015, Seoul, South Korea
* Joint first authors
PDF | Show BibTex | Show Details