MosaicMVS: Mosaic-based Omnidirectional Multi-view Stereo for Indoor Scenes

Paper

Code

Supplementary

Abstract

overview

We present MosaicMVS, a novel learning-based depth estimation framework for a mosaic-based omnidirectional multi-view stereo (MVS) camera setup. It uses a regular field of view (FOV) MVS network for an omnidirectional imaging setup with explicit consideration of hypothetical voxel-wise FOV overlaps. The resulting depth predictions are accurate and agree on the omnidirectional multi-view geometry. Unlike existing MVS setups, MosaicMVS camera setup can be easily applied to omnidirectional indoor scenes without having to account for constraints such as intricate epipolar constraints and the distortion of omnidirectional cameras. We validate the effectiveness of our framework on a new challenging indoor dataset in terms of depth estimation, reconstruction, and view synthesis. Our framework outperforms the state-of-the-art methods in a large margin in all test scenes.


Depth estimation showcases

From posed RGBs with divergent camera setup (left), our framework estimates accurate depth maps (right).


Depth map comparison

Comparison of estimated depth maps. Comparing the baseline result (left) and ours (right), depth maps produced by our pipeline clearly show fewer artifacts.


Reconstruction showcases


Poisson reconstructions with the depth maps produced via COLMAP.


Poisson reconstructions with the depth maps produced via our pipeline. Zoom in by scrolling. You can toggle the “Single Sided” option in Model Inspector (pressing I key) to enable back-face culling (see through walls). Select “Matcap” to inspect the geometry without textures.


Pipeline overview

MosaicMVS Architechture

Overview of the proposed MosaicMVS. For every target-source pair, the estimated camera pose and depth values, and source view indices are pre-processed and they were used to infer depth maps. The estimated depth and confidence map were obtained by constructing valid view cost volume regularization in 3D convolution and they were filtered using the pre-processed data without depth values.


Valid view cost volume overview

valid view cost volume

The construction process of the valid view cost volume ${{C}}_{V}$. From ${O}_{sum}$, we adaptively change the denominator ${O}_{filled}$ of the cost metric per hypothetical voxel based on each warped feature volume $\tilde{{F}}_i(v)$. Also, we minimize the effect of zero overlapping hypothetical voxels in ${O}_{filled}$.



Citation

@article{,
  title={{MosaicMVS}: : Mosaic-based Omnidirectional Multi-view Stereo for Indoor Scenes},
  author={},
  journal={},
  year={2022}
}