Milestone2

April 1 - 8, 2024

New Progress

Based on playtest results, we found that users want to be able to interact with virtual objects and have them naturally coexist with the real objects and environment. So this week, we focused on three features to address these needs:

  1. Changing textures on walls, floors, and ceilings
  2. Adding objects into the room
  3. Fusing real-world objects with the virtual environment
Video Demo of RenovAR on Quest 3

Challenges with Passthrough

The first two features, raycasting and object placement, are relatively straightforward because we can treat the scanned room as a virtual environment. These features work similarly to how they function in VR, with controller input being used to interact with the environment. However, we encountered difficulties with the third feature: displaying real objects in the scene. After extensive research, we discovered that the Passthrough API provided by Meta currently only offers coarse-grained control, such as underlaying or overlaying the virtual scene. While real objects like furniture can be scanned using the API, we found that enabling floor or wall texture overlays would directly cover the passthrough video feed. This means that the overlays are not seamlessly integrated into the 3D environment as we initially thought.

Fortunately, Quinten Dauwe, a fellow Mixed Reality developer from the Black Whale Studio's XR Dev Discord, kindly pointed us towards Selective Passthrough. We had previously watched many of the studio's high-quality MR tutorials for the Meta Presence API and joined their Discord channel after attempting to tweak the default passthrough shader without success. With Quinten's guidance, we were able to selectively display scanned furniture in passthrough, although the textures are still overlaid on top of the passthrough feed.

Plan for Next Week

In the upcoming week, we plan to:

  1. Explore Meta's Mixed Reality Utility Kit (MRUK), which Quinten mentioned.

    2. We discovered that this kit not only creates convenient abstractions over the Scene API but also provides ways to test our MR apps in the Unity editor with 30 pre-made room prefabs. Over the next week, we will be porting existing logic from the Scene API to the MRUK. We hope that MRUK will offer a better way to mix passthrough with overlay textures.

  2. Develop a new passthrough window feature

    3. This feature will allow users to seamlessly view their backyard or outdoor surroundings from within the virtual space. It aims to bridge the gap between the real and virtual worlds by providing users with a simultaneous view of both environments.

    The introduction of a passthrough window is expected to significantly enhance the user experience and provide a more comprehensive perspective during the interior design process. By incorporating the real outdoor scenery, users can better assess how their interior design choices interact with and complement the exterior view. This holistic approach enables homeowners and designers to make more informed decisions, ensuring that the interior design harmonizes with the surrounding environment.

    Currently, most 3D renderings created by interior designers rely on artificial or generic outdoor scenery when depicting windows. However, the RenovAR team recognizes that the outside view is often just as crucial as the interior design and should be given more consideration during the planning phase. By leveraging the passthrough window feature, RenovAR aims to provide a more realistic and personalized design experience, setting itself apart from traditional rendering methods.