VisionOS App Development: The Era of Spatial Computing
Table of contents
- What is the Apple Vision Pro?
- The potential of visionOS development
- Transforming Industries
- What is spatial computing?
- The concepts of spatial computing
- Immersive learning: AR and VR
- The 2024 AR/VR market
- Developing your first visionOS app
- How Magora can help
- Conclusion
The Apple Vision Pro, a device seamlessly merging digital content with our physical surroundings through the visionary visionOS software, is shaking up our expectation of technology: reimagining what we can do – and how we interact – with it.
This article explores the transformative impact of the Vision Pro and the opportunities it presents for brands and developers, from redefining entertainment with high-resolution displays to enhancing education through interactive learning environments: excitement, innovation, and exploration.
As developers and enthusiasts embrace the potential of spatial computing, we welcome a new dawn of immersion and interaction.
Source: Apple
What is the Apple Vision Pro?
The Apple Vision Pro must first be described as a significant advancement in computing technology; a 'spatial computer' that seamlessly integrates digital content and applications into the user's physical environment. Utilising a combination of eye movements, hand gestures, and voice commands, users can effortlessly navigate through apps by simply looking at them and selecting with a tap of the fingers.
Operating on the visionOS software platform, the Apple Vision Pro offers access to a range of built-in applications such as Apple TV and Safari, along with the ability to connect calls via FaceTime and download third-party apps from the App Store. Recent developments include the introduction of over 600 new apps specifically tailored for the Apple Vision Pro, further expanding its functionality within the extensive app ecosystem across iOS and iPadOS.
One of the key features of the device is its 3D interface, which liberates apps from the constraints of traditional displays, providing users with an expansive canvas for multitasking and collaboration. Additionally, users have the option to pair a Magic Keyboard and Magic Trackpad or utilise the Mac Virtual Display feature to create a customised workspace tailored to their preferences.
What can it do?
Let’s drill down into some of the ways the Vision Pro can impact your day-to-day life (and recreation time!)
Sport
The Apple Vision Pro boasts 'ultra-high-resolution displays that deliver more pixels than a 4K TV for each eye', allowing users to view content 'on a 100-foot screen'. Enhancing the experience, apps like PGA TOUR Vision lets users experience sporting events with features such as ‘real-time shot tracking layered on top of 3D models of real golf courses’, accompanied by key statistics. This capability provides an immersive experience that brings the tour to life within the user's space.
Source: Apple
Entertainment
Feature films and television series have capitalised on Apple's innovative technology, with platforms like Disney+ and other prominent entertainment apps optimising their content for the Vision Pro. IMAX offers an exceptional viewing experience for both 2D and 3D content, featuring renowned documentaries such as Deep Sky in IMAX's expanded aspect ratio. Additionally, select programming from Warner Bros. is ‘available in 4K resolution and Spatial Audio with Dolby Atmos’.
Source: Apple
Apple has also made significant strides in this arena, offering over ‘200 3D movies’ through Apple TV+. Moreover, Apple Immersive Video immerses viewers in the action with ‘180-degree, 8K 3D recordings captured with Spatial Audio’.
Music
Apple Music provides users with an enhanced auditory experience through Spatial Audio featuring Dolby Atmos and Lossless Audio with ultra-low latency. This feature is compatible with AirPods Pro (2nd generation) with USB-C.
Apple Vision Pro users can access their preferred video and music streaming platforms via Safari too, unlocking various features, including multitasking with multiple windows, immersive full-screen viewing, Spatial Audio playback, and the option to ‘launch an Environment’ offering users an unparalleled entertainment experience.
Source: Apple
Mental health
The Apple Vision Pro delivers a selection of immersive apps aimed at promoting mindfulness and relaxation. These include Healium, offering serene escapes into nature; Lungy, facilitating interactive breathing exercises and active meditation sessions; and Odio, which envelops users in personalised spatial soundscapes.
The potential of visionOS development
Apple's visionOS enables and encourages innovative app development experiences that are tailored specifically for the Vision Pro, presenting an exciting opportunity to shake up the way we work, play, and consume content.
With the recent introduction of the Vision Pro to the market, developers now have the opportunity to pioneer new pathways in this evolving field. The integration of virtual and augmented reality expands the possibilities beyond conventional screens, offering users immersive experiences through thoughtfully crafted apps and environments.
The arrival of this technology creates avenues for traditional apps to undergo significant transformations too, adapting to leverage the expansive capabilities of VisionOS. This evolution breathes new life into familiar applications, aligning them with the vast potential offered by this pioneering platform.
Immersion
VisionOS liberates users from the limitations of traditional mobile operating systems, providing an immersive app interaction experience. This innovative approach enables seamless transition between various engagement modes, including windowed interfaces, augmented reality (AR) overlays seamlessly integrated with the real world, and full immersion within virtual environments created by the app.
Spatial Computing
With visionOS's robust support for spatial computing, apps can leverage the user's physical environment and movements.
Users can interact with 3D objects by extending their arms or experience virtual environments like art galleries with a sense of presence. The possibilities for spatial interaction are vast, offering a new frontier in user engagement.
Familiarity
For developers proficient in Apple's suite of tools and frameworks like SwiftUI, visionOS offers a smooth transition. Utilising familiar resources streamlines the development process, empowering creators to focus on crafting customised and immersive experiences aligned with the platform's capabilities.
Accessibility Emphasis
Central to visionOS is its commitment to accessibility. By prioritising inclusive design principles, the platform empowers users to interact with apps through various modalities, including eye gaze, voice commands, and alternative pointers driven by head and hand movements. This focused approach not only broadens the platform's accessibility but also fosters a culture of inclusive app development, ensuring that technology remains accessible to all users.
Transforming Industries
Vision Pro has the power to transform operations, processes, procedures and investigation for a wealth of sectors, from education and healthcare to retail and automotive.
Let’s take a look at some of them.
Education
Students can immerse themselves in historical events in 3D, conduct virtual dissections in biology classes, or simulate complex surgeries in a risk-free virtual environment, fostering deeper engagement and understanding among learners.
Healthcare
Surgeons can undergo enhanced training, visualise crucial data during procedures, and collaborate with remote specialists in real-time through AR overlays.
In mental health, AR therapy sessions could be developed to address phobias, anxiety disorders, and facilitate rehabilitation processes, while disease detection becomes more efficient – ‘Google is continuing to update their AR microscope repository on GitHub, allowing developers and healthcare professionals to use augmented reality to better detect cancerous cells.’
Retail
Widely reported, in the retail industry AR is redefining the shopping journey. Customers can now virtually try on apparel, visualise furniture in their home, or explore virtual product showrooms.
In the health and beauty sector, e.l.f. Cosmetics and Google’s futuristic campaign garnered attention. Teaming up to produce ‘interactive YouTube advertisements’, viewers can ‘click on the ad to start up an AR experience where they can see how the cosmetics look on their faces.’
Design and Engineering
Architects and engineers can create and manipulate 3D models in real-time, collaborate with peers virtually, and conduct immersive virtual tours of planned structures or infrastructure projects, streamlining the design and development process.
Manufacturing and Maintenance
For technicians in manufacturing and maintenance sectors, Vision Pro facilitates remote troubleshooting, provides access to manuals and instructions through AR overlays, and enables visualisation of data and schematics, enhancing efficiency and accuracy in complex machinery operations, while ‘digital twins and simulated experiences can help workers understand and respond to serious situations in an immersive environment.’
AR can also help with maintenance, highlighting those elements of a machine that require a service, and facilitate remote help, ‘off-site engineers leverage AR to highlight and point to elements in 3D space to help end users solve problems without dispatching a field technician.’
Automotive
AR overlays can expedite maintenance and repair tasks by projecting labels and information directly onto car parts and visualising complex engine data to diagnose issues more efficiently, streamlining the troubleshooting process and enhancing overall productivity.
Furthermore, AR is assisting drivers, from locating parking spots to keeping tired folks awake through facial recognition.
‘Why Everyone is Wrong about the Apple Vision Pro’, the pros, cons, and experience: “The age of spatial computing is coming, but is that future now? How will it impact our lives? And what’s it’s REALLY like?”
What is spatial computing?
The term is contentious, with many failing to agree on what the term “spatial computing” actually means. Labels range from Mixed Reality (MR) or Extended Reality (XR) to AR, the metaverse or holographic computing.
The Washington Post describes it as ‘an immersive video feed of the physical world plus the internet…you’re not seeing the real world. You’re seeing a nearly live streaming video of your living room or kitchen with apps superimposed on there.’
However you’d like to define it – for the purpose of this article, we’ll call it spatial computing – these are the core concepts.
The concepts of spatial computing
Labelled by Apple as the building blocks of spatial computing in visionOS, windows, volumes, and spaces are the concepts used to create your app and define how the user interacts with it.
Let’s look closer.
Windows
Built with SwiftUI, windows contain ‘traditional views and controls’ – the user can ‘create one or more windows in [the] visionOS app’ before adding ‘depth to [the] experience by adding 3D content.’
Volumes
3D volume adds ‘depth to your app’ – they are ‘SwiftUI scenes that can showcase 3D content using RealityKit or Unity, creating experiences that are viewable from any angle in the Shared Space or an app’s Full Space.’
Spaces
Apps, by default, are launched into the Vision Pro’s Shared Space side by side. The Apps can ‘use windows and volumes to show content, and the user can reposition these elements wherever they like.’ If the user wants to create ‘a more immersive experience, an app can open a dedicated Full Space where only that app’s content will appear. Inside a Full Space, an app can use windows and volumes, create unbounded 3D content, open a portal to a different world, or even fully immerse people in an environment.’
The 2024 AR/VR market
In 2024, the AR and VR market is forecasted to generate a whopping US$38.6 billion in revenue and ‘grow at an annual rate of 10.77% (CAGR 2024-2028), resulting in a projected market volume of US$58.1bn by 2028.’
AR Software emerges as the largest segment, expected to command a market volume of US$13.0 billion in 2024 alone, while the global user base is projected to balloon, reaching an impressive 3,674.0 million users by the end of 2028.
What trends can we expect to see?
From the metaverse and avatars to wearables and – you guessed it – Apple Vision Pro, we can expect our technological world to be rocked by 2024’s AR trends.
Let’s take a closer look.
Metaverse
We can expect to see the refinement and enhancement of avatars, popular in the social media sphere – named Spatial Personas, i.e. realistic avatars, they ‘use machine learning to help display more body language and face tracking than ever before for social AR/VR interactions for FaceTime.’
Mobile AR
With an extensive range of use, from gaming to business, mobile app AR is being used for all manner of tasks including navigation, remote assistance, and product visualisation.
WebAR
Accessible and universal – simple ‘AR experiences are provided through a web browser compatible with most devices with a camera’ – WebAR can provide ‘face filters [and change the] colour of hair or objects, background replacement, and simple 3D objects on just about any device regardless of hardware.’
Wearables
With the Apple Vision Pro and Meta’s competing products, Quest 3 and Quest Pro, marking ‘a significant milestone in the road to the future of augmented reality headsets for consumers’ we’re noticing ‘a new era of interfaces’ and immersive experiences rethinking the way we experience and operate technology.
Navigation
AR’s blend of reality and technology draws the users’ eyes up from the screen and straight ahead with ‘3D markers and guides’ transposed onto actual surroundings used to guide the walker to their destination.
Human-centred experience
VisionOS emphasises the design of apps that prioritise the user. The immersive experience offered by the Apple Vision Pro lends itself to intuitive and inclusive experiences and meaningful, interactive collaboration unmatched by traditional web and mobile apps. Developers are charged with making the user’s experience otherworldly.
Let’s take a look at some popular examples…
This AR app identifies stars and constellations visible in the night sky by pointing your phone camera towards them.
Snapchat’s AR additions make photos and videos little more fun thereby enhancing the social media experience.
This VR rhythm game lets players slash coloured blocks to the beat of music.
Virtually travel the world from the comfort of your couch, exploring landmarks and locations at your leisure.
Experience historical events or scientific phenomena first-hand through VR simulations.
Developing your first visionOS app
If this article has inspired you to start building your own visionOS app using SwiftUI, then the following guide may come in handy.
visionOS development begins with ‘Xcode, which supports the visionOS SDK.’
Note: the below numbered points are highlights taken from Apple’s guide
1. Place content onscreen using scenes.
A scene contains the views and controls to display onscreen. Scenes also define the appearance of those views and controls when they appear onscreen. In visionOS, you can include both 2D and 3D views in the same scene, and you can present those views in a window or as part of the person’s surroundings.
2. Create your Xcode project
Choose File > New > Project. Navigate to the visionOS section of the template chooser and choose the App template. When prompted, specify a name for your project along with other options.
Include a Reality Composer Pro project file when you want to create 3D assets or scenes to display from your app.
3. Modify the existing window
Build your initial interface using standard SwiftUI views. Views provide the basic content for your interface, and you customise the appearance and behaviour of them using SwiftUI modifiers. For example, the .background modifier adds a partially transparent tint colour behind your content.
4. Handle events in your views
Many SwiftUI views handle interactions automatically — all you do is provide code to run when the interactions occur. You can also add SwiftUI gesture recognisers to a view to handle tap, long-press, drag, rotate, and zoom gestures. The system automatically maps the following types of input to your SwiftUI event-handling code.
Indirect input. The person’s eyes indicate the target of an interaction. To start the interaction, the person touches their thumb and forefinger together on one or both hands. Additional finger and hand movements define the gesture type.
Direct input. When a person’s finger occupies the same space as an onscreen item, the system reports an interaction. Additional finger and hand movements define the gesture type.
Keyboard input. People can use a connected mouse, trackpad, or keyboard to interact with items, trigger menu commands, and perform gestures.
5. Build and run your app in Simulator to see how it looks.
The visionOS simulator has a virtual background as the backdrop for your app’s content. Use your keyboard and your mouse or trackpad to navigate around the environment and interact with your app.
Tap and drag the window bar below your app’s content to reposition the window in the environment. Move the pointer over the circle next to the window bar to reveal the window’s close button. Move the cursor to one of the window’s corners to turn the window bar into a resizing control.
How Magora can help
With one finger on the pulse and a keen eye for the next revolutionary tech product on the market, Magora has hit the ground running with Apple’s Vision Pro.
Well versed in its capabilities and possibilities, our extensive mixed reality development and virtual reality development experience suits this new release perfectly – we have already developed exciting apps with more in the works. Build a powerful visionOS app with us.
Find out how we can help your Vision Pro ambitions. From discovery and user research to business planning, we have you (thoroughly) covered.
Conclusion
We are on the cusp of a new technological era with the Apple Vision Pro, merging digital content with physical space to reshape our interactions and perceptions.
From revolutionising entertainment and education to transforming healthcare and industry, visionOS app development offers vast possibilities. As developers push innovation boundaries, we anticipate numerous immersive experiences captivating users worldwide.
Beyond technology, human imagination and ingenuity shape our world. Using spatial computing's capabilities to create experiences that exceed the ordinary, we unlock boundless potential for innovation and exploration.
As we venture into spatial computing, the convergence of digital and physical realms, we welcome a future where the unimaginable becomes reality.
Team Lead/ Senior Software Engineer
related
recent
recommended
