Mixed reality combines VR, AR and AI to blend the digital world with the physical.
Mixed-Reality (MR) is a new computing platform that‘s emerging from the current fields of Augmented- and Virtual-Reality (AR/VR). MR blends the physical and the digital worlds into a single space, using a combination of cutting edge optical hardware and artificial intelligence (AI) software. The device leading the current wave in the MR revolution is the Microsoft HoloLens.
MR combines the immersive capabilities of VR (a head-mounted optical device rather than a small mobile phone window) with AR’s ability to bring data into the physical world. This powerful combination makes it a medium suited to both consumer and enterprise domains and marks a new paradigm in the way people interact with an increasingly data-rich world.
Why is MR Important?
The ability to blend digital information and objects seamlessly into our physical world creates a huge potential step change in the way we use computers. Firstly, and potentially most disruptively, MR will make physical screens a thing of the past. Want a 60 inch plasma screen on your wall? Just put it there. Now want that same display pocket sized and mobile? No problem.
The second thing MR will change is how we collaborate. The natural way for humans to interact is to gather around together in a shared space, to talk and to gesture. Traditional 2D screens and video conferencing are a massive step backwards in terms of communication and understanding, but when it comes to digital information they’ve been our only option until now. MR allows us to collaborate both locally and remotely around digital objects and data using the same human tools we use for physical objects. These could be interactive architectural models, medical scan data, or 3D data visualisations.
Finally, by scanning the world around us, MR devices can superimpose data in position over the top of objects or physical spaces. This capability has huge implications for empowering workers in complex, data-rich environments such as engineering, complex manufacturing or emergency response.
Mixed Reality Today
The flagship of current-generation MR devices is the Microsoft HoloLens. This tech is capable of rendering full 3D objects and data as holograms, and of understanding the world around it in order to place those holograms in physical space. The HoloLens is a self-contained Windows 10 computer that doesn’t require wires or a network connection to function. This makes it just as capable on a factory floor, or the middle of a construction site – as it is in an office.
With the current generation’s $3000 price point, it is more likely that the next generation of MR devices will revolutionise the consumer market. Meanwhile, industries as diverse as transportation, healthcare, industrial design and education are currently adopting MR to revolutionise their business processes.
Scenarios include:
- Visualising complex projects, processes and designs prior to signoff
- Training distributed workforces on complex tasks
- Showing data in situ to facilitate real-time decision making
To flesh out these ideas, here are several case studies.
Thyssenkrupp
Multinational elevator firm Thyssenkrupp are an early adopter who use HoloLens for remote engineering challenges. Thyssenkrupp have a 20,000-strong global workforce of field engineers who deal with a diverse range of complex products and installations. Each site visit presents a unique challenge and delivering timely solutions is both critical to budgets and to corporate reputation.
An MR solution to these problems empowers this workforce by providing remote assistance to first diagnose issues and then provide realtime, ‘heads-up’ guidance. MR also allows product and situation-specific information & diagnostics to be overlaid in the engineer’s view.
These two capabilities together reduce on-site time, increase success rates, and increase customer confidence in the quality and safety of Thyssenkrupp’s services and products. Although similar functionality could have been achieved with traditional screen-based remote support apps, Thyssenkrupp found that MR’s ability to blend data, 3D technical models and video into the real world around an engineer provided unparalleled benefits to success rates.
Volvo
Volvo, a company with very different challenges, have deployed HoloLens across many of their business processes. In vehicle design they have used HoloLens to collaboratively pre-visualise 1:1 scale designs, an approach that is less expensive, more powerful and more collaborative than the alternative approaches of a computer screen, or a clay or 3D printed physical model.
At the manufacturing stage they have used MR to provide workers with heads-up views on their tasks and on the functioning of the production line, allowing quicker training, improved safety, and fewer delays. And at the sales stage, Volvo have deployed HoloLens to help customers understand product features, and to view vehicle customisations in realtime, increasing both conversion and satisfaction.
Volvo’s experience is a great example of how MR’s disruptive potential is not limited to a single business function but – like mobile or internet technology before it – will potentially disrupt every step of the value chain and the consumer environment in which it operates.
Steps to successful MR projects
When considering your organisation’s first steps into MR – be that conceiving internal pilot projects, or searching for a delivery partner – there are some key considerations to ensure a good outcome.
First, you must find the right niche. Not every single business function will have potential for disruption by MR in the short term, but in any business category there will be plenty of low hanging fruit. Understanding the capabilities that MR unlocks will help identify these. Crucially, lumping MR in with other ‘immersive’ tech such as VR is unlikely to be the best approach. While VR is great for deeply isolated, immersive activities (e.g. a designer visualising the interior of an apartment), MR’s strengths speak more to spaces which are about collaborating with others, understanding the as-built environment, or augmenting real world activities with real-time data.
Secondly, think about the MR user experience. In the long term, this means having in-house talent or external partners who understand the medium. Many great web design agencies struggled in the early years to convert that experience to smartphone app design; the same will be true in the AR/MR space. The UX toolbox for MR is still being built as designers and agencies learn what works – one project at a time. It’s also important to pay attention to the short term though. As with any new medium, users will need more hand-holding in an MR experience than they would in an equivalent app or web experience. A great on-boarding experience is therefore crucial to the wider user experience.
Another skill set that’s critical is the ability to optimise for the current generation of devices. While the HoloLens is powerful, it’s also a mobile device with a standalone windows PC embedded inside it. Unlike VR systems which can rely on a hyper-powerful desktop physically attached, the HoloLens must do everything in head-mounted hardware. This requires development talent well attuned to the issues of creating optimised, performant content.
Finally, once you’ve identified a great niche, nailed the user experience, and polished the look and performance of your concept, it’s incredibly important to communicate that concept to the wider world. Perhaps you can get 5% of stakeholders in your organisation to try out the experience in the HoloLens itself, but for the other 95%, all they get to see is someone staring into thin air.
MR Videos, which show both the real world user and the holographic content, in a single view, are the most popular solution to this problem – creating something which is at once magical and sharable. Microsoft has their own proprietary solutions for this problem, as do a small number of other accredited agencies.
Rob Minson, CTO at Fracture Reality
Image Credit: Halfpoint / Shutterstock