SGAligner 📐

3D Scene Alignment with Scene Graphs

ICCV 2023

1ETH Zurich, 2Microsoft Mixed Reality & AI Labs
Paper Code Dataset Benchmark Video Poster

SGAligner aligns 3D scene graphs of environments using multi-modal learning and leverages the output for the downstream task of 3D point cloud registration, 3D point cloud mosaicking, and 3D alignment of a point cloud in a larger map that contains changes

News 📰

  • Sep 2023: Video presenting SGAligner released.
  • Jul 2023: SGAligner accepted to ICCV '23.
  • May 2023: SGAligner preprint released on arXiv.
  • Apr 2023: Code released.


Building 3D scene graphs has recently emerged as a topic in scene representation for several embodied AI applications to represent the world in a structured and rich manner. With their increased use in solving downstream tasks (eg, navigation and room rearrangement), can we leverage and recycle them for creating 3D maps of environments, a pivotal step in agent operation? We focus on the fundamental problem of aligning pairs of 3D scene graphs whose overlap can range from zero to partial and can contain arbitrary changes.

We propose SGAligner, the first method for aligning pairs of 3D scene graphs that is robust to in-the-wild scenarios (ie, unknown overlap -- if any -- and changes in the environment). We get inspired by multi-modality knowledge graphs and use contrastive learning to learn a joint, multi-modal embedding space. We evaluate on the 3RScan dataset and further showcase that our method can be used for estimating the transformation between pairs of 3D scenes. Since benchmarks for these tasks are missing, we create them on this dataset. The code, benchmark, and trained models are available on the project website.

Video 🎬

Architecture Overview

Visual Outputs of Downstream Applications

Conference Poster

References 📚

[1] Wald et. al, RIO: 3D Object Instance Re-Localization in Changing Indoor Environments, ICCV 2019
[2] Wald et. al, Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions, CVPR 2020

BibTeX 🙏

          title={SGAligner : 3D Scene Alignment with Scene Graphs}, 
          author={Sayan Deb Sarkar and Ondrej Miksik and Marc Pollefeys and Daniel Barath and Iro Armeni},
          journal={Proceedings of the IEEE International Conference on Computer Vision (ICCV)},