Join me in exploring XR Development for my final Master of Educational Technology Directed Studies, and finding ways of making XR Development more manageable and accessible for other educators, students and creatives navigating XR and beginning to develop their own projects.
One of my main goals in embarking upon my Masters of Educational Technology was to explore newer-to-me forms of storytelling that could help to bring education to life. Over the past 7 years as I’ve worked my way through my Masters, being introduced to UBC’s Emerging Media Lab and the work they are doing there, having been approached by a board member from the Juno Beach Centre to explore newer ways of sharing stories from World War II that would connect with Canadian school children, travelling to Normandy to visit the museum and see how they are sharing stories there and identifying the gaps in their storytelling, and being shortlisted and interviewed for an interactive and immersive storytelling fellowship in Norway and exploring the stories I might wish to create for that fellowship, the idea of delving deeper into education and storytelling through XR – Extended Reality, teased at my synapses. I took an early course in VR in Education, attended XR symposiums and showcases, began researching and exploring different applications of XR in education and storytelling, collaborated on an open course in Immersive Experiences in Natural and Cultural History Education, developed an idea for an AR storytelling app to share natural and cultural history experiences, and am working my way through other people’s courses in XR, including the University of Michigan’s Extended Reality series of courses. While I am only now beginning to write about all of what I have learned here, the series of posts that follow come from research and a rabbit hole that I began to travel through 6-years-ago, and to date have included four talks at national and international conferences in education, technology, and storytelling, sharing my findings and explorations, and one talk to a Journalism Masters Class at the American University of Cairo. I have shared that last talk below as it shares many of the goals that I set out to accomplish with this directed studies.
Defining Immersive and Interactive Media
Before we get into where my interests and observations are taking me with my final directed studies for my Masters, I thought we’d explore the terminology around immersive media, experiential media, and extended reality, as there is ambiguity around the terminology, so by addressing this at the onset, I hope to at least make it clear as to how I am using the terminology.
Immersive and Interactive Media / Experiential Media
Immersive and Interactive Media are basically the current term for what was termed transmedia, referring to media that is immersive and interactive in nature, allowing those that are engaged with it to experience it in a way that allows them to step into the experience and / or become a part of the media and / or interact with the media, potentially having an impact on how the experience is shaped.
Examples of forms of immersive and interactive media include virtual reality, augmented reality, escape rooms, 360 video, choose your own adventure stories, alternate reality games … and the list goes on. Another name for this form of media is experiential media
Extended Reality (XR)
Extended Reality is a form of immersive / experiential media that takes users into a new reality or places virtual objects into a user’s real world, enhancing everyday life with technology. Basically making impossible sensorial experiences, possible. (Hargreave, 2021a)
Included within extended reality are virtual reality (VR), augmented reality (AR), and mixed reality (MR).
Virtual Reality (VR)
Virtual Reality is a computer-generated environment with scenes and objects that appear to be real, making the user feel they are immersed in their surroundings. (Iberdrola, n.d.) While this often is experienced through the use of a headset, a user could also walk into a virtual reality environment created by projectors and screens. This simulated environment and experience could be similar to or completely different from that of the real world in which a user is physically situated.
Examples of virtual reality include 360 video, when experienced through a headset, meeting in Mozilla Hubs, and the Immersive Van Gogh Exhibit.
Augmented Reality (AR)
Augmented Reality (AR) is an interactive experience where the real, physical world is digitally augmented. The augmentation is usually visual in current applications, but it can also be through other senses, such as auditory or haptic. (Wong, 2021)
Examples of augmented reality include Pokemon Go, the ReBlink exhibition at the AGO, and selfie masking filters in SnapChat.
Mixed Reality (MR)
There is a lot of debate around what constitutes Mixed Reality. I have always thought of it and defined it as allowing for real and virtual elements to interact with one another, and for users to interact with virtual elements in a similar way they would in the real world or at least in a similar way that they might interact with a touch screen. (Hargreave, 2021b)
Others have defined mix reality (Speicher et al, 2019) as:
- a synonym for augmented reality
- a combination of AR and VR
- a stronger version of AR
- a type of collaboration
- the idea of aligning physical virtual environments.
- the spectrum from the real environment without any augmentation to the virtual environment, which is completely synthesized computer-generated virtual content.
This is further confused by some researchers only working within the theoretical and not experimenting with the tools in real time and some corporations attempting to use the term ‘mixed reality’ to market the next level advances in extended reality.
Due to all the debate around defining mixed reality, and based on one research paper, the unlikelihood of a consensus between the various experts in research and industry, I will likely avoid the term mixed reality and instead describe the characteristics of various extended reality applications, and possibly where they lie on the reality–virtuality continuum developed by Paul Milgram and colleagues in 1994.
Reality-Virtuality Continuum
The Reality-Virtuality Continuum as proposed by Paul Milgram, Haruo Takemura, Akira Utsumi, and Fumio Kishino in their 1994 paper, encompasses all possible variations and compositions of real and virtual objects. They hypothesized that everything between real and the virtual is mixed reality, and that mixed reality makes up both augmented reality and augment virtuality. (Milgram et al, 1994) Basically suggesting that mixed reality is the blending of the physical and the virtual world with varying degrees of augmentation.
The problem with the RV Continuum in its current form is that it only takes into account visual, not sound, olfactory or haptics senses. (Speicher et al, 2019) It ultimately needs to be redeviced with how technologies are evolving.
Augmented Virtuality
Augmented Virtuality refers to perceiving mostly virtual content, while still seeing some real world content. (Nebeling, 2022)
My Interests and Explorations in Extended Reality
Virtual Reality
I have long been fascinated with extended reality, ever since I was first introduced to the concept of virtual reality in an episode of Murder She Wrote in 1993. It excited me to imagine how extended reality could be used to create immersive story experience and bring worlds to life that are created in one’s imagination. The problem for me though was that whenever I tried it, which I did over the years whenever an opportunity present itself, was that it was too much sensory overload for me, causing mr to feel motion sick (which I now know is cybersickness) and sparked migraines. (Weech et al, 2019)
Persistence pays off though, as in 2018 I finally had a VR experience that did not spark cybersickness or a migraine. The experience in question was a virtual plank walk designed to help people overcome their fear of heights. (Basbasse et al, 2023) In exploring the features that made the difference in this experience for me and talking to one of the designers of the experience, I believe the quality of the filming / graphics made the main difference in this experience, as well as perhaps the experience being designed to limit your time in VR. This also got me inquiring from the designer if VR was being used in other ways in the medical world, and if perhaps it was being used with pain management. My interest? By this point I’d been living with chronic pain following two car accidents for 5 years, and was curious if perhaps there were ways this technology could be used to help people better manage their pain. This sparked me to begin explore different medical avenues and uses to VR, as I began to contemplate what might be useful to me and to the community of chronic pain and fatigue sufferers that I now belonged to, many of whom are predominantly home bound.
Augmented Reality
I have always loved and been drawn to educational experiences that incorporate storytelling and encourage the imagination. This was developed in me as a parks naturalist and is part of how I approached science education, and encourage kids to think, imagine, experience and explore.
In approaching work and storytelling, post the car accidents that very much changed my life, how I work, what I can handle and in some ways who I am, I have been increasingly drawn back towards natural and cultural history in my storytelling, as nature soothes my symptoms. When I travel I also love learning and love to envelop myself in the story of a space, but thanks to my post accidents realities I cannot always handle busy museums with noise, lights, and other sensory triggers. This got me imagining, first with my fellowship interview in Norway and later as I explored the Juno Beach Centre in Normandy, if I could create mobile storytelling experiences outside of the museums in places of natural and cultural history, where people could learn and enjoy the stories of the place as they explored outdoors. In both cases, as they were looking to me to imagine newer ways of bringing such stories to life, I began to imagine this with augmented reality as experienced through our mobile devices. This made me smile, as not only did this idea hold the potential for learning and storytelling, but also some of the fun, whimsy and magic that augmented reality brings with it. This is an idea that I further began to sketch out on my travels and in my ETEC 522 Course – Ventures in Educational Technology, and that I am working towards making a reality.
My Goals with this Directed Studies into Extended Reality
I have a tendency to dream big and my dyslexic brain sees connections and expanding and overlapping storyworlds. Usually this means that I do not have the money and often all the knowledge for orchestrating all that I imagine at the get go. My dyslexia has taught me to problem solve though, and life and an imagination and brain that don’t let go of ideas have taught me patience and to play the long game, approaching my ideas in digestible pieces. Thats what I have been doing with this directed studies. I essentially started the work on it after the first year into my Masters, slowly taking advantage of research experiences as they became available, participating in XR symposiums and showcases when they present themselves, taking formal and informal courses in XR, accepting opportunities to speak on what I am working towards with XR and learning along the way, and beginning to flush out my ideas and pitch them to funding bodies.
For this directed studies, my goals are to:
- Write up the VR case studies that are helping me to identify the strengths and challenges in VR for patient communities.
- Write up the AR case studies that are helping me to identify the strengths and challenges in AR for natural and history education.
- Begin to develop and further experiment with VR and AR development.
- Sharing what I have learned to help make it more manageable and accessible for other educators, students and creatives navigating XR and begin to develop their own projects.
A Beginning
While in some ways I see this directed studies as an end to my researching and a beginning to my developing, there will always be more research to be done in seeing how others are developing and applying the XR, especially as the technology is still developing and despite its decades of history, still very much in its early days for development, application and adoption.
Development-wise, waiting on gear and difficulties with my health and post concussion syndrome, have very much slowed down my intended goals with development and where I’d like to be at this stage, in terms of developing my own XR applications in VR for patients and AR for natural and cultural history. However, as my post accidents self has learned, pushing through when my health issues becomes problematic, only making things worse and setting me back further. Meaning that I have had to be patient with myself, have had to get my health back to a stable place (during a year that posed a lot of challenges with that), and have had to address the backlog of pressing deadlines from my health going sideways before I’ve been able to engage in development. I’ve put in the work to do that, and am now in a good and healthy space to embark on development. This is much later than I would have liked, but that is not something I can change with my newer disability realities post the car accidents. As such, my initial development goals for this directed studies will be smaller for now, and I will continue working on the bigger goals, post the directed studies and will continue to write those up as I do.
This also means that for the Guide to XR Development that I will be developing to make XR Development more manageable and accessible for other educators, students and creatives, this will similarly take shape in stages and over time, some coming sooner with my directed studies and some continuing over time, as I further delve into XR development and experiments of my own.
Thank you for joining me on this journey into XR Development, and being patient with me as I take the time to do this in a way that is healthy for me with my health and disability realities. Hopefully my journey as the tortoise on this road will inspire others to explore and embark on such a journey of their own, no matter at what speed and working with whatever challenges your life and situation might pose along the way.
References
El Basbasse, Y., Packheiser, J., Peterburs, J., Maymon, C., Güntürkün, O., Grimshaw, G., & Ocklenburg, S. (2023). Walk the plank! Using mobile electroencephalography to investigate emotional lateralization of immersive fear in virtual reality. Royal Society open science, 10(5), 221239. https://doi.org/10.1098/rsos.221239
Hargreave, E. (2021a, November 15). An introduction to immersive experiences in natural & cultural history education – StoryToGo Classroom. StoryToGo Classroom. https://storytogo.ca/classroom/course/immersive-experiences-in-natural-and-cultural-history/lessons/intro/
Hargreave, E. (2021b, November 15). Mixed Reality (MR) in Natural & Cultural History Education – StoryToGo Classroom. StoryToGo Classroom. https://storytogo.ca/classroom/course/immersive-experiences-in-natural-and-cultural-history/lessons/mr-in-natural-cultural-history-education/
Hargreave, E., [Stories, EdTech & Digital Media with Ahimsa Media]. (2020, November 29). Story Steppers – an AR storytelling app to the natural & cultural history of a place [Video]. YouTube. https://www.youtube.com/watch?v=2xeWjBWYcAI
Hargreave, E., [StoryToGo]. (2023, December 10). Exploring Immersive Media to Develop Creative & Engaging Stories with Master of Journalism Students [Video]. YouTube. https://www.youtube.com/watch?v=Ff1cMMiH5cc
Iberdrola. (n.d.). Virtual Reality: another world within sight. Retrieved December 7, 2023, from https://www.iberdrola.com/innovation/virtual-reality
P. Milgram, H. Takemura, A. Utsumi, F. Kishino: Augmented Reality: A class of displays on the reality-virtuality continuum. In Proceedings of Telemanipulator and Telepresence Technologies, 1994.
Nebeling, M. (2022, June 2). Intro to AR/VR/MR/XR: Technologies, applications & issues. Coursera. https://www.coursera.org/learn/intro-augmented-virtual-mixed-extended-reality-technologies-applications-issues
M. Speicher, B.D. Hall, M. Nebeling: What is Mixed Reality? In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI 2019, Glasgow, Scotland, UK, May 4-9, 2019.
Weech, S., Kenny, S., & Barnett-Cowan, M. (2019). Presence and Cybersickness in Virtual Reality Are Negatively Related: A Review. Frontiers in psychology, 10, 158. https://doi.org/10.3389/fpsyg.2019.00158
Wong, Y. (2021, November 15). Augmented Reality (AR) in Natural & Cultural History Education – StoryToGo Classroom. StoryToGo Classroom. https://storytogo.ca/classroom/course/immersive-experiences-in-natural-and-cultural-history/lessons/ar/
