A series of reports in The Information paints a detailed picture of the progress, policies and challenges Apple’s plan to develop a virtual, augmented or mixed reality headset has faced since the initiative gained momentum back in 2015.
Citing several people familiar with the product, including those who directly worked on it, the reports describe a contest of desires over the direction of the device. The confrontation took place between Apple’s mixed reality product development team (referred to as the “Technology Development Team”) and celebrity Apple designer Jony Ive and his industrial design team. The report sheds light on Apple’s direction for the device, which Bloomberg recently reported is nearing release.
Apple’s mixed reality efforts began almost by accident when the company acquired a German AR startup called Metaio to use some of its technology in Project Titan, its self-driving car project, The Information sources say. Another key milestone was that Apple hired Mike Rockwell, AR/VR project team leader, from Dolby Laboratories. Starting in 2015, Rockwell built a team that included Metaio co-founder Peter Meyer and Apple Watch manager Fletcher Rothkopf.
In 2016, several demonstrations of augmented reality were shown to Apple board members. In one, a tiny Triceratops grew to life-size in front of the board members. In another, the room has become immersive foliage. But the board was not Rockwell, but the most significant barrier to the company. According to the information, it was Ive, who led the industrial design and user interface teams at Apple.
Ive and his team opposed the VR helmet because they believe that virtual reality separates users from people and the world around them, and virtual reality helmets look unfashionable. But the technology development team enlisted the help of the industrial design team by introducing a concept: an outward-facing screen on the front of the headset that displayed the wearer’s facial expressions and eyes to those around them. The owner could see the surrounding people through the external camera channel.
Rockwell and colleagues developed and released ARKit in 2017, an app development suite that allowed developers to create augmented reality apps for the iPhone and iPad using technologies and techniques that could later be adapted for the headset.
The decision to make the headset a standalone device reportedly caused major problems. For example, some felt that the best way to make it work well would be to put more features on a single chip. But since the silicon work had already been done, they needed to find ways to deal with the latency associated with multiple chips in a device communicating with each other. They also created the software with the assumption that the base station plan would come to fruition.
However, the device is in the final stages of development. Bloomberg said last week that an enhanced version of the product was recently presented to Apple’s board of directors and that Apple has “accelerated”development of the headset’s software, which is a fork of iOS called rOS. (R stands for “reality”.)
A report from The Information reveals many details about the upcoming headset. It will have a resolution of at least 4K per eye, which the team believes is the minimum to keep users from perceiving the image as pixelated, unlike most current consumer VR headsets. Its embedded processor will be closely related to the M2 processor, which is expected to appear on Macs and iPads in the coming months.
Due to the limited computing power of the M2 chip in a headset without a tethered base station (the canceled base station is said to have a super-high performance M1 Ultra), the avatars will be cartoonish. The Information sources also say attempts were made to create more photorealistic avatars when the base station was part of the plan, but the uncanny valley was a problem.
Apple originally planned to release the headset in 2019, but now it looks like it could be announced either later this year or 2023. In addition, Apple plans to introduce more natural augmented reality glasses as the next product, but this device may still be years before it ships.