Multi-user, Scalable 3D Object Detection in AR Cloud

Siddharth Choudhary

Nitesh Sekhar

Siddharth Mahendran

Prateek Singhal

Magic Leap

CVPR 2020 Workshop on Computer Vision for AR/VR

[Paper]

[Supplementary]

[Slides]

[Video]

Screenshot of the output of our system running on Magicverse AR Cloud and displayed through Magic Leap 1 AR device.

Abstract

As AR Cloud gains importance, one key challenge is large scale, multi-user 3D object detection. Current approaches typically focus on the single-room, single-user scenarios. In this work, we present an approach for multi-user and scalable 3D object detection, based on distributed data association and fusion. We use an off-the-shelf detector to detect object instances in 2D and then combine them in 3D, per object while allowing asynchronous updates to the map. The distributed data association and fusion allows us to scale the detection to a large number of users concurrently, while maintaining a lower memory footprint without loss in accuracy. We show empirical results, where the distributed and centralized approaches achieve comparable accuracy on the ScanNet dataset while reducing the memory consumption by a factor of 15.

Paper

S. Choudhary, N. Sekhar, S. Mahendran, P. Singhal.

Multi-user, Scalable 3D Object Detection in AR Cloud.

In CVPR Workshop on Computer Vision for Augmented and Virtual Reality, Seattle, WA, 2020.

[bibtex]

Shipped on Magic Leap 1

A new experimental feature in the @magicleap OS 0.98.10 update; Found Objects. Developers can access this information which makes me want to make a bridge-building game across my living room. #magicleapdevs. pic.twitter.com/Zc9aSVz3dE
— Josh Naylor (@JoshNaylor) March 30, 2020

Acknowledgements

This work was possible due to contributions from an extended team of software engineers and researchers including but not limited to Arumugam Kalaikannan, Nitin Bansal, Manushree Gangwar, Shiyu Dong, Divya Ramnath, Khushi Gupta and Adithya Rao.