Evaluating Alternatives to SFM Point Cloud Initialization for Gaussian Splatting

Yalda Foroutan1   Daniel Rebain2   Kwang Moo Yi2   Andrea Tagliasacchi1,3,4

1Simon Fraser University, 2University of British Columbia, 3University of Toronto, 4Google DeepMind

Paper Code

Abstract

3D Gaussian Splatting has recently been embraced as a versatile and effective method for scene reconstruction and novel view synthesis, owing to its high-quality results and compatibility with hardware rasterization. Despite its advantages, Gaussian Splatting's reliance on high-quality point cloud initialization by Structure-from-Motion (SFM) algorithms is a significant limitation to be overcome. To this end, we investigate various initialization strategies for Gaussian Splatting and delve into how volumetric reconstructions from Neural Radiance Fields (NeRF) can be utilized to bypass the dependency on SFM data. Our findings demonstrate that random initialization can perform much better if carefully designed and that by employing a combination of improved initialization strategies and structure distillation from low-cost NeRF models, it is possible to achieve equivalent results, or at times even superior, to those obtained from SFM initialization.

Descriptive text for image
COLMAP (Left) vs. NeRF-based (Right) initialization point clouds for Gaussian Splatting. The NeRF-based initialization provides a much more complete model of the scene structure, while also being faster to construct with posed images.

Visualization

Random Initialization COLMAP Initialization Random Initialization + Depth Loss NerfAcc Initialization + Depth Loss
Random Initialization COLMAP Initialization Random Initialization + Depth Loss NerfAcc Initialization + Depth Loss
Random Initialization COLMAP Initialization Random Initialization + Depth Loss NerfAcc Initialization + Depth Loss
Random Initialization COLMAP Initialization Random Initialization + Depth Loss NerfAcc Initialization + Depth Loss
Random Initialization COLMAP Initialization Random Initialization + Depth Loss NerfAcc Initialization + Depth Loss
Random Initialization COLMAP Initialization Random Initialization + Depth Loss NerfAcc Initialization + Depth Loss

Acknowledgements

This work was supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant [2023-05617], NSERC Collaborative Research and Development Grant, the SFU Visual Computing Research Chair, Google, Digital Research Alliance of Canada, and Advanced Research Computing at the University of British Columbia.

BibTeX

@INPROCEEDINGS{Foroutan2024arxiv,
          author = {Yalda Foroutan, Daniel Rebain, Kwang Moo Yi, Andrea Tagliasacchi},
          title = {Evaluating Alternatives to SFM Point Cloud Initialization for Gaussian Splatting},
          booktitle = {arxiv},
          year = {2024}
        }