News

Apple and Facebook Compete to Create a More Natural AR/VR Experience

May 06, 2021 by Jake Hertz

Apple’s new patent introduces its spin on bringing interactive touch control to AR/VR. Meanwhile, Facebook is going another direction.

One of AR/VR limitations as it currently stands is the unnatural and unintuitive way a user's hands interact with the mixed reality environment. Some devices use hand gesture tracking, some require you to use a physical controller, but none are quite intuitive or comfortable yet. 

 

Example of an HTC Vive, which uses a hand controller for interaction.

Example of an HTC Vive, which uses a hand controller for interaction. Image [modified] used courtesy of HTC

 

The sense of touch in AR/VR systems has been a significant research problem that companies and academics have invested a lot of time and money into solving. 

Apple has been teasing their AR/VR equipment for some time now, and recent news shows that they, too, have been looking for ways to address the touch problem. Last week, Apple revealed a new patent focusing on interactive touch in AR/VR, which further cements its investment in the technology. 

This article discusses the concept behind their new patent and how it differs from Facebook's recently announced direction for bringing touch control to AR/VR. 

 

Apple’s New Patent 

In 2015, Apple acquired AR startup Metaio, a company that invented what they termed "Thermal Touch." The technology operates on the principle that when people physically touch objects, they heat up, even in tiny amounts. 

Using information from both an infrared camera and a standard camera, Thermal Touch can observe changes in the thermal profile of the environment, allowing for any physical surface to become a touchscreen. Though Apple patented this technology close to the acquisition, it has been kept under wraps for quite a while.

Now, as of April 22, the company has successfully received a "continuation-in-part" application, which essentially reveals more information on the subject matter, but is also in many ways a repetition of the previous application. With recent funding announced yesterday, there is potential for this technology to become more realized. 

 

Apple's thermal touch proposal utilizes infrared to track touch on physical objects viewed through an AR interface. Image from the US Patent & Trademark Office

 

The new patent seems to further cement this technology as part of Apple's AR/VR arsenal and likely to be a feature on their headset whenever it is to be released. 

The implications of the technology could be transformative for AR by accurately portraying the physical world digitally. Furthermore, the interaction would be done more naturally for humans, working solely on natural touch and requiring no hardware on or in hand.

Though Apple is one of the more prominent names trying to get involved in the AR/VR universe, Facebook is also trying to up the ante.  

 

Facebook’s Take on Touch 

Facebook is another technology giant that has been active in this space for years and continues to innovate. In late March, the company unveiled to the public what their direction was for the future of touch and control in AR/VR, which takes a different approach than Apple.

Facebook plans to have users wear a specialized strap on their wrist to control and manipulate their virtual environment. The strap claims to use electromyography (EMG), in which sensors translate electrical signals from motor nerves into digital commands that the system can interpret and act upon. Facebook claims that its solution is so effective that it can interpret finger movement down to a millimeter, making the system essentially effortless to use––only requiring the slightest motion to perform a virtual click.

 

EMG wrist strap concept from Facebook.

EMG wrist strap concept from Facebook. Screenshot used courtesy of Facebook

 

The company also sees this as only the first step in the long journey of potential functionality that the wrist strap can provide. Eventually, they anticipate being able to touch and move virtual objects and objects at a distance using the same method.

Beyond that, the company plans on eventually implementing haptics into this system. Sending vibrations and feedback to the user would allow for interaction with virtual objects to feel as if they were physical objects, making touch control more intuitive and natural. 

 

Moving Forward (in Several Directions)

While each company is taking different approaches towards touch and control within the AR/VR realm, both are innovative and full of potential. There is hope to see success from both companies as they both fight to finally push the technology over the cusp and into mainstream adoption.

 


 

Interested in learning more about recent AR/VR innovations? Read more down below.

Featured screenshot used courtesy of Facebook.