Mixed Reality (MR) in Natural & Cultural History Education
As so often occurs with emerging technologies, there is a period of debate around the use of terms. Mixed Reality (MR) is one such word that is going through those growing pains at present. Mixed Reality is getting used by some (such as Mozilla and Microsoft) as the umbrella term for XR, interchangeable with Immersive Media, Immersive Experiences, and Experiential Media. Developers however define Mixed Reality as a type of XR.
It is the context of the developers’ definition of Mixed Reality that we will be using here.
Defining Mixed Reality
Mixed reality is the newest immersive technology. It allows real and virtual elements to interact with one another, and for users to interact with virtual elements in a similar way they would in the real world or at least in a similar way that they might interact with a touch screen.
Due to the virtual world being projected on to the real world, some people consider mixed reality to be augmented reality 2.0, with the added ability of being able to interact with and engage with the virtual elements in the real world.
Augmented or Mixed Reality
As immersive experiences and technologies are changing so quickly many people get confused as to the differences between virtual, augmented, and mixed reality – and particularly between augmented and mixed reality.
How would you classify these experiences?
You’d think this would be straight forward, but it is not always. Have a look at the immersive experiences below. How would you classify each? As augmented reality, mixed reality or both?
Our apologies that these are not experiences that we can bring into your living room at present, but that is alas one limitation with mixed reality at present, especially if you do not have a device of your own.
Tremona-Castello Archaeological Park
HoloBrain
Imagined Odyssey
Rhyality Immersive Hall
Spatial
Visiting London’s Courtauld Gallery and the British Museum from your Living Room
Dynamic 365 Guides
Planet Ice: Mysteries of the Ice Ages at the Canadian Museum of Nature
HoloAnatomy
Reflecting on one of the immersive experiences above, answer the following questions on the padlet discussion wall below:
- what appealed to you?
- what would you change?
- would this work for you as an educator? why or why not?
- what would the challenges be?
- what would the opportunities be?
A Closer Exploration of Two of the Experiences – the Challenges & Opportunities
Below are two immersive experiences that Erica has had and an exploration of challenges and opportunities that exist with each.
Tremona-Castello Archaeological Park
Erica partook in the immersive media experience at Tremona-Castello Archaeological Park in April 2019. While the experience holds a lot of potential for a bit of magical time travel, it is not quite there yet, and is at present dealing with some of the challenging growing pains of immersive media.
Erica was fortunate on her visit to talk to one of the visionary’s behind the project, Nadia Lupi, and learn about the problem they were looking to solve with an immersive media experience and the challenges that they needed to work around in creating an immersive eperience.
What?
Tremona-Castello Archaeological Park is a 13-Century Medieval Settlement found on a hillside near the base of Mount Saint Giorgio in Tremona, Switzerland. While an open air heritage museum site, Tremona-Castello is also an active archaeological dig site.
Problem Being Solved
To take visitors into the past, teaching them about daily life in a medieval village, the Archaeological Park and local tourism office wanted to turn the site into a living history museum and recreate some of the buildings there.
Unfortunately, they were not granted funding for a living history museum and village recreation; however this did spark the idea for some virtual travels through time and the virtual recreation of the village instead.
The Vision
The vision for this immersive experience is to use augmented reality to overlap a virtual 3D model of the village over the archaeological remains of the village, and see people within the village going about their daily lives.
The Present Reality
At present, the immersive experience at Tremona-Castello Archaeological Park is an augmented reality experience involving a prerecorded audio tour and static 3-D models of a few medieval scenes within the archaeological park, as seen through immersive glasses.
Challenges with the Present Reality
- the immersive experience is quite limited and not representative of the full 3D model of the village, as illustrated in the above video
- aside from wisps of smoke, everything in the medieval scenes are static
- the immersive glasses are uncomfortable (especially for someone wearing glasses) and finicky
- the screen within the immersive glasses can be difficult to see at times
- once the audio is started you have to keep pace with it, and cannot replay sections – this means you can’t stop to read interpretive signage and on a hot day, you cannot escape from the sun to go have a shade break
- the need for the immersive glasses to enjoy the experience, limits access to these stories to when the visitor’s centre is open
Erica’s Dreams For
Personally, Erica thinks that the present experience at Tremona-Castello Archaeological Park would be better enjoyed without the immersive glasses, but rather viewed through an app on the visitor’s smartphone or tablet. This would allow visitors to take in the experience at their own pace and would allow them to enjoy it even when the visitor’s centre is closed. If creating such an app for this augmented experience, the app could also include other augmented experiences from the region – like an interpretive fossil walk of Mount Saint Giorgio, or a wine tour of the region with augmented reality vitoculturalist walking visitors through the grape growing and wine making process. This could even become part of broader series of augmented reality walking tours throughout Switzerland.
For Tremona, one issue with the above idea is cellular coverage, as Tremona, Switzerland borders Northern Italy and as result people run into issues of their phones switching cell networks with the bordering country. Erica feels though that there may be a Proximity-Based Learning solution to this, as she has used a smartphone audio travel guide app in Maui (Gyspy Guide) that tells stories based on location, even when her phone is in airplane mode, provided you started the app while connected to wifi.
Erica feels this could also be transformed into a mixed reality experience that could be quite magical, if the medieval village was represented with the ghostly holograms illustrated in the video, and visitors could talk to them, ask them questions, and see them physically at work. It would also be cool, if in those spots that villagers were at work on something, if there was a way visitors could take part too and learn from them – either physically creating or doing something in the real world or through the manipulation of virtual objects.
Of course, many of the ideas shared in the above paragraph would also be cool if you could see, communicate with, and engage with through your smartphone or tablet.
Do you have others ideas for Tremona-Castello Archaeological Park? Tell us about them below on the padlet below.
HoloBrain
What?
The HoloBrain is a holographic depiction of a human brain that guides students through a 3 dimensional exploration of the brain. You can read more about the HoloBrain’s initial vision here.
Problem Being Solved
Designed to help medical students to understand the complex spatial relationship of parts of the brain. Visualizing 3D structures through 2D MRI imaging is difficult.
Initial Challenges
- to access the HoloBrain you need access to HoloLens Glasses and you need the HoloBrain software, limiting access for most to the UBC campus and to scheduled times that compete with others who may wish to access
- the HoloLens glasses are uncomfortable (especially for someone wearing glasses) and finicky
- even regular users of the HoloLens glasses and the HoloBrain struggle with issues of responsiveness to hand gestures
- the screen within the HoloLens glasses can be difficult to see at times
- hygiene and safety in the present pandemic
- virtual learning and inaccessibility in the present pandemic
How the Holobrain Team is Addressing Those Challenges
One of the HoloBrain teams future goals is to be able to allow people from around the world to be able to interact with the HoloBrain in real time. This has begun, in response to COVID-19, leading to the development of methodology to allow for the easily recording and broadcasting of a HoloBrain lesson.
To solve this problem, the SpectatorBrain – an app that synchronously mirrors the visualization of the HoloBrain with a HoloLens on a mobile device for recording and broadcasting – was created, allowing instructors to record HoloBrain lessons and make them widely available for students. The plan for the Spectator Brain once students are again learning on-campus, will be to allow students to synchronously see the HoloBrain on their own devices in a live lecture.
With the recent release of Hololens 2 with the Microsoft Mixed Reality Toolkit (MRTK), HoloBrain 2.0 will be developed, creating new ways for students to touch, grab and isolate brain structures. With this UBC will be developing an augmented-reality neuroanatomy lab that students will be able to access any time to study with 3D virtual specimens.
There is also a HoloBrain WedVR app in the works that you can read more about here: HoloBrain in WebVR.
Do you have others ideas for HoloBrain? Tell us about them below on the padlet below.
Being Aware of the Present Challenges
Due to the muddled use of the term ‘mixed reality’ in the present immersive technology marketplace, it can make things very confusing for consumers, as there are a number of occluded headsets, meaning headsets that are specifically designed for virtual reality and do not provide a view of the real world or augmented reality, that term themselves as mixed reality headsets. Generally headsets describing themselves as ‘Windows Mixed Reality headsets’ are occluded.
As our purposes here is to explore mixed reality as we initially defined – where the virtual elements are overlaid upon the real world, and the user is able to interact with those virtual elements in the real world – it is the headsets that allow users to do that we will focus on here. Often these headsets are termed ‘glasses’.
Solution Providers
There are a number of companies who are working on mixed reality technology in the form of mixed reality initially defined, including HoloLens (Microsoft), Magic Leap, Rokid, Varjo XR-1, Vuzix M-Series, Moverio (Epson), X2 (ThirdEye), Norm, NReal, and Vuzix Blade. Also worthy of mention is Mojo Vision, the first smart contact lens.
The figures in the chart below were collected in October 2020:
Challenges
Many of the challenges with mixed reality at present are tied to these immersive technologies, including:
- comfort (especially for eye glass wearers)
- battery life
- responsiveness
- ease of seeing the holograms through the glasses (this has been an issue with the HoloLens)
- cost
- health concerns (around sharing devices)
- which technology do you choose? will this limit which mixed reality experiences you and your students have access to?
At present HoloLens appears to be the frontrunner in education, as it has been investing in it, with research and development of mixed reality experiences both at the post-secondary and grade school level.
What other challenges strike you in using mixed reality in education or at natural and cultural history sites?
Contemplating the Opportunities in Education and Natural and Cultural History
Given the above, there are opportunities in creating more comfortable immersive technologies and one’s that are more responsive. Already companies are beginning to come out with such products, including glasses that resemble traditional glasses and even contact lenses. Personally, as an eyeglass wearer, I’d love to see more ways of creating mixed reality that don’t involve glasses or contact lenses at all – more of the sorts of opportunities which can be gear independent, like Spatial, the Planet Ice: Mysteries of the Ice Ages, and the installations at the Rhyality Immersive Hall. Software like TouchDesigner even offer the opportunity for educators to build interactive environments in their classrooms or at their cultural sites without the expensive headsets.
As for what is possible, take a look at this educational tease from Microsoft …
Some classrooms and schools have even turned their students into the developers of mixed reality …
If experimenting with developing mixed reality on your own or with your students is of interest to you, here are a few places that you can get started: