How Apple Vision Pro could benefit STEM research
Apple has recently announced its first spatial computing device, the Apple Vision Pro, a headset that promises to “seamlessly [blend] digital content with the physical world”1—a technology referred to as mixed reality. The device, which is expected to be available in early 2024, promises to deliver extraordinary new possibilities for entertainment and productivity. But what does it mean for researchers, especially those in STEM fields? In this post, I would like to explore some potential benefits that Apple Vision Pro or similar devices could bring to scientific research and education.
What is Apple Vision Pro?
Vision Pro is a standalone headset, much like conventional virtual reality headsets such as those offered by Oculus. Vision Pro will run its own iOS-derived operating system designed for augmented reality software. The headset contains high-resolution displays powered by custom Apple chips that promise to generate real-time high-quality visuals. The device will also feature a 3D camera that can capture spatial photos and videos in 3D and with surround sound.
Vision Pro’s main appeal is its three-dimensional interface that frees apps from the traditional two-dimensional plane, allowing applications to appear side by side at any scale, overlaid over the real world. Users will be able to interact with digital content using eye tracking, hand gestures, and their voice, as well as traditional input methods like a mouse and keyboard. As part of the Apple ecosystem, it will also users to stream content from their Mac or iPhone wirelessly to use their Vision Pro like a huge, private, and portable display.
Why may Apple Vision Pro be useful for researchers?
It’s obvious why gamers, digital artists, and movie fans are excited about Vision Pro, but what promises does it hold for researchers? While it hasn’t attracted widespread attention, scientists have already been making use of consumer virtual reality headsets in their research2.
Automated image analysis and data interpretation
Vision Pro could be integrated with computer vision and machine learning algorithms to enable researchers to perform image analysis tasks more efficiently and accurately. For example, researchers could use Vision Pro to perform real-time analysis of histology slides, identifying features of interest, measuring features, and annotating results. AI image analysis has already been demonstrated and used in pathology laboratories3, and Vision Pro could allow researchers and clinicians to use such technologies to make real-time analyses in their acquisition workflow.
Data visualization
Mixed reality can allow researchers to visualize their data in more intuitive and interactive ways. For example, researchers could use Vision Pro to create 3D graphs, charts, maps, or diagrams that can be manipulated using voice commands or gestures that allow users to interact with data as if it were a real world object4. Vision Pro could also help researchers display multiple data sets simultaneously, overlay data on physical objects or environments, or create immersive data environments that can be explored from different perspectives.
Precision-based work
Vision Pro could enable researchers to perform precision work more accurately and safely using mixed reality applications. For example, researchers could use Vision Pro to simulate experiments or procedures that require high levels of accuracy or dexterity, such as surgery, engineering design, or nanotechnology fabrication. Various mixed, augmented, and virtual reality have already been used to guide complex surgical procedures5.
In the classroom
Vision Pro could use mixed reality applications to enhance the teaching and learning experience. Researchers could use Vision Pro to create engaging and interactive lectures or demonstrations that can be delivered remotely or in person. Vision Pro could also help instructors provide feedback or guidance to students using eye tracking or voice input. In some pilot programs, augmented reality has been used to create rich and immersive educational content that can supplement conventional learning methods6.
Conclusion
Can the Apple Vision Pro become the first successful mixed reality product? Or is it doomed to end up as another failed attempt at the technology à la Google Glass?7 Since its unveiling, Vision Pro has been the target both of excitement and ridicule8, but Apple has impressive history of turning what seem like novelties into the new paradigm, just as System 1.0, iPhone, and iPad were the first major success stories in GUI software, touch-screen phones, and standalone tablets, respectively. While the device is not yet publicly available, Vision Pro and similar devices could be a disruptive new technology. It may be worth giving some thought about the possibilities of using augmented reality in your own research.
References
1. Introducing Apple Vision Pro: Apple’s first spatial computer - Apple. https://www.apple.com/newsroom/2023/06/introducing-apple-vision-pro/.
2. Matthews, D. Virtual-reality applications give science a new dimension. Nature 557, 127–128 (2018).
3. Rakha, E. A. et al. Current and future applications of artificial intelligence in pathology: a clinical perspective. J. Clin. Pathol. 74, 409–414 (2021).
4. Data visualization via VR and AR: How we’ll interact with tomorrow’s data. ZDNET https://www.zdnet.com/article/data-visualization-via-vr-and-ar-how-well-interact-with-tomorrows-data/.
5. Sakai, D. et al. Augmented, virtual and mixed reality in spinal surgery: A real-world experience. J. Orthop. Surg. Hong Kong 28, 2309499020952698 (2020).
6. Volioti, C. et al. Augmented Reality Applications for Learning Geography in Primary Education. Appl. Syst. Innov. 5, 111 (2022).
7. Why Google Glass Broke - The New York Times. https://www.nytimes.com/2015/02/05/style/why-google-glass-broke.html.
8. Apple’s Vision Pro Announcement Met With Memes And Mockery. https://www.forbes.com/sites/danidiplacido/2023/06/06/apples-vision-pro-announcement-met-with-memes-and-mockery/?sh=3013fdbd2084.
Comments
You're looking to give wings to your academic career and publication journey. We like that!
Why don't we give you complete access! Create a free account and get unlimited access to all resources & a vibrant researcher community.
