StoryToGo

Exploring storytelling in its many forms in this digital age

Connect with our Community

EdTech

Conceptualising an Extended Reality Open Education Guide

by 6 Comments

As we step into 2024 and I wrap up my ETEC 580 Directed Studies and Master of Educational Technology, I want to take a moment to conceptualise the Extended Reality Open Education Guide that I have begun to build; to give it shape, intention and create a road map for it, as well to seek feedback from you, our readers, as I continue to further my journey, experiments, and goals in extended reality (XR) beyond my Masters. As I previously wrote in Exploring XR Development with my Final Master of Educational Technology Directed Studies, the work I have done through the MET Program at UBC is a beginning to this work in XR development.

Here’s where I am at present in mapping out where I am going with this work:

XR Open Education Guide for StoryToGo

As I continue to develop an XR Open Education Guide on StoryToGo, much as I did with the Web Monetization Guide, there are 4 main areas that I will be focusing on, developing, and building upon over time:

  • XR Case Studies
  • XR Development Experiments and Process Blogging
  • XR Development Series of Mini-Courses
  • Addressing Accessibility in Extended Reality

XR Case Studies

The goal of the Extended Reality Case Studies is to explore different extended reality experiences and applications to discover what resonates with myself and my team and to identify the challenges. By writing these up as case studies, the hope is to create a resource for our team in developing extended reality experiences, and to share that resource more broadly with other independent creatives, educators and students wanting to explore extended reality. Secondarily, these XR Case Studies will also be used as resources and examples within the XR Development Series of Mini-Courses.

At present, these XR Case Studies are focused around augmented reality (AR) natural and cultural history experiences and virtual reality (VR) health and wellness experiences, as these are the areas that my team and I are currently developing extended reality projects. Over times, we will broaden these case studies to other areas of interest.

Extended Reality Case Studies

As my team’s and my current priority is on AR natural and cultural history experiences, the current case studies include:

I am also writing up a case study on the TaleBlazer AR Game at the Royal Botanical Garden’s Rose Garden at present.

Other XR Case Studies on the current ‘to write’ list, include:

  • Notre-Dame de Paris, The Augmented Exhibition
  • Plank Walk VR
  • Pain Distraction VR
  • Concussion Therapy VR

As the XR Case Studies grow in number on StoryToGo, we will create an Experiential Reality Library, like the Web Monetized Content Library that we’ve begun to create.

XR Development Experiments and Process Blogging

As an indie creative who likes experimenting with emerging ways of crafting stories, I find it helpful to experiment with the technology for building those stories. That way even if we contract a larger team on our builds, I know what I am talking about and know what is possible. Often as I dive into such experiments, I also learn that building stories in these newer ways with emerging technologies is not so daunting, cost prohibitive, and inaccessible as people think. Thus, as I create and experiment with extended reality development, using different extended reality development tools and platforms, I intend to create process blogs walking people through the ‘how to’ of my experiments creating with different technologies.

The purpose?

  • to create a running log / record of the process to pull from for the XR Development Series of Mini Courses;
  • to engage others around this process, and crowdsource ideas and solutions; and
  • inspire others to experiment with their own XR builds, and provide them with a resource to helping them do so.
Extended Reality Development Experiments and Process Blogging

Currently my goals for my own projects involve Unreal Engine and motion capture technology. This, however, is not the most accessible entry point for educators and students wanting to begin their own experiments with XR, both in terms of the learning curve and in terms of the expense of the technology needed for. As such, I decided for the purposes of creating a more accessible entry to extended reality for teachers and students, that I want to begin with some more accessible extended reality builds / development ideas, and then build towards the more involved pieces using UnReal Engine and Unity, as part of my long term, on-going work.

What this means, is that my initial AR build will be with TaleBlazer to create some early development options within the open Extended Reality Course that are designed for teachers and students, and then later build Units on developing with UnReal Engine and Unity, as well as other more accessible extended reality development solutions, like Stornaway.io, CoSpaces, Halo AR, polycam, MyWebAR, and 8th Wall. If you have other suggestions of XR development tools and platforms that my team and I should be experimenting with, please let me know about them in the comments on this post.

In terms of the extended reality projects that my team and I have been developing, they include:

  • WWII stories from my friend Manami Calvo (Saito)’s family from their experience as Canadians of Japanese ancestry, living on the West Coast in WWII
    • our plan for this is to create some initial experimental ‘art installations’ using Web AR, that are accessible via smartphones and tablets
    • we plan on creating these utilizing animated ‘ghostly figures’ created with Unreal Engine and a motion capture suit, along with historic photos and video, as well as recorded interviews
    • we will then use those art installations to go after proper funding to further development

  • health / wellness VR Experiences
    • our plan is to start with a kids yoga class in VR that can be used in hospitals, as well as at home
    • create in Unreal Engine with a motion capture suit
    • the yoga will be lead by Lori Yearwood (wearing the motion capture suit) and will involve her transforming to different plants and animals with different poses
    • the children will also be able to select the background world (from a variety of calming choices) for their yoga experience

XR Development Series of Mini-Courses

To make things as easy as possible for other indie creatives, educators and students to begin to explore and develop their own extended reality projects, I intend to take what I have learned through my extended reality research, case studies, experiments and process blogging, and build a series of Extended Reality Mini-Courses for people to use as a guide in creating their own extended reality projects. The goal here being to remove the fear of extended reality being a far reaching, inaccessible goal by building their understanding of extended reality, getting them experimenting with it and envisioning the possibilities, and sharing with them a variety of pathways to creating extended reality, including step-by-step guides.

Extended Reality Development Courses

In developing a series of Extended Reality Mini-Courses, I am exploring other Extended Reality Learning Resources to see what is currently available to those wishing to explore extended reality further, so as best to identify a different, unique approach to the subject and include any beneficial resources that are currently missing.

The Extended Reality Learning Resources that I am currently delving into, include:

Pre-Existing Extended Reality Courses

In approaching the extended reality course development for the StoryToGo Classroom site, I have come to the conclusion that XR development is more manageably approached as a series of mini-courses. Currently, I am thinking in terms of building the following three courses:

  • First Steps in Extended Reality Development – An Introduction
  • Extended Reality Development
  • Extended Reality User Testing and Refining

The talk I gave for the class at the American University in Cairo follows the layout for the first mini-course: 

My current breakdown for ‘First Steps in Extended Reality Development – An Introduction’ will include:

  • What Extended Reality Is
    • types of Extended Reality
    • applications of Extended Reality
    • delivery of Extended Reality and Devices

  • Extended Reality Case Studies
    • an informal introduction to user testing
    • creating mini case studies in user testing extended reality experiences, and recording stars and wishes from the experiences
  • Developing an Extended Reality Idea of Your Own
    • Why XR?
      • what type of XR?
      • is this the strongest choice?
      • how do you wish to deliver? to which devices? why?
    • Do you need to create all at once or can you create it in accessible steps?
    • Steps in creating
    • Creating a roadmap

Then the Extended Reality Development Course will be a continual work in progress, adding different Units, covering how to build XR with different extended reality development solutions, starting with TaleBlazer.

Extended Reality Mini Course to Develop for StoryToGo

If you have suggestions for the structure of or things I should be including in the XR Development Series of Mini-Courses, please share your thoughts in the comments.

Addressing Accessibility in Extended Reality

While not a standalone piece on my brainstorming document, for those of you who have begun to read our XR Case Studies, you may have recognized that accessibility in extended reality is central to my team’s and my explorations and development goals. This is very much driven by various disabilities and health and wellness issues that my team and I have experienced, but is also an aspect of extended reality design and user testing that has not been discussed and addressed as well as it should yet.

For now, my team and I intended to address Accessibility in Extended Reality throughout our case studies, process blogging, and course materials. I also had an invitation to come on as an advisor on accessibility to UBC’s Emerging Media Lab, after I brought accessibility and my perspective on with a few of the students at their Autumn Showcase. This is an invitation I intend to follow up on this year, with the hope that between that and my own team’s process thinking through accessibility in extended reality, as we share extended reality case studies, work on our own extended reality development and process blog on it, and create our Extended Reality Development Series of Mini-Courses, that we also begin to build a standalone piece or pieces that address accessibility in extended reality. Not sure what that will be yet, but if you have suggestions for this, I’d love to hear about them in the comments.

Your Thoughts?

I’d love to hear your thoughts in comments below on the direction I am taking with this Extended Reality Open Education Guide. If you have suggestions for future XR Case Studies, on XR Development Tools and Platforms to try, the structure of or things I should be including in the XR Development Series of Mini-Courses, or ways to further address Accessibility in Extended Reality, please share them in the comments. Thank you!

Also, if anyone wishes to become involved in any of the above endeavours with myself and my team, please reach out and we can chat further.

Thank Yous

Thank You!

The work that I have begun here has been inspired by and is evolving, thanks to:

  • Dr David Vogt for encouraging my dreaming, entrepreneurial thinking, and edtech product design and development;
  • Dr David Porter in exploring open education and searching for sustainable funding pathways for;
  • Saeed Dyanatkar for seeing my potential in dreaming up different pathways for storytelling and education, always making time for me when I had questions, and making me feel welcome and valued whenever I wandered into the Emerging Media Lab;
  • Dr Kyle Stooshnov and Juliana Loh for creating a framework for my first steps into thinking about XR development and experimenting with 360 video;
  • Dr Jennifer Jenson and Dr Suzanne de Castell for fostering my game design thinking;
  • Dr Heidi Janz, Dr Michelle Stack and Dr Paul Hamilton for helping me to process my newer disabilities, frame disabilities and myself as disabled in my storytelling, and begin to understand how to advocate for disability and accessibility in education (and feel empowered to do so, understanding it as a strength, as oppose to something to hide);
  • Yvonne Dawydiak for demonstrating the alternative pathways to navigating UBC, approaching teaching creatively in a hands-on way, and an empathy driven approach and practice;
  • Lori Yearwood for her vision in beginning to create case studies here on StoryToGo to act as resources for our students, workshop participants, clients and partners, and the broader independent creative and education community – for encouraging me to begin to share my disabilities in our storytelling and work, and sharing her own wellness challenges as we began to build our health, wellness and accessibility workshops, resources, and storytelling – and for always being a willing and enthusiastic supporter and co-creator on the ideas that wander through my mind; and
  • Manami Calvo for seeing the potential for sharing her family’s WWII stories in AR, joining me in crafting experiments around, and in trusting me with her family’s stories.

Also an enormous thank you to the Interledger Foundation for helping fund my research and work on Sustainable Funding Solutions for Creatives and Educators, and to CNIE – RCIÉ and BCIT who have helped fund my research and coursework throughout my Masters.

Thank you all, for all your help, support, direction and inspiration!

Seeing the Invisible Augmented Reality Art Exhibit – an Extended Reality Case Study

by 4 Comments

In this second case study in our series exploring extended reality (XR), we examine the Seeing the Invisible Augmented Reality (AR) Art Exhibit, as exhibited by the Royal Botanical Gardens. This AR art exhibition featuring the art of 13 artists from around the world was initiated by the Jerusalem Botanical Gardens and displayed concurrently at 12 Botanical Gardens worldwide.

Project:

Seeing the Invisible Augmented Reality Art Exhibit

Initiated By:

Jerusalem Botanical Gardens in partnership with Outset Contemporary Art Fund, with the support of the Jerusalem Foundation

Curated By:

Hadas Maor and Tal Michael Haring

Host At:

Royal Botanical Gardens (RBG)

Type of Experience:

Augmented Reality (AR) Outdoor Art Exhibition

Device for Experiencing:

Smartphone or Tablet, preferably with headphones

Link to Project:

https://seeingtheinvisible.art/

Date Partook in the Experience:

September 18, 2022

Country of Origin:

Israel

Country Where I Experienced:

Canada

Reflections On My Experience

Interview Between Tucson Botanical Gardens Executive Director Michelle Conklin & Seeing The Invisible Curators Hadas Maor & Tal Michael Haring

The Project:

In 2021, as the world continued to face the pandemic and many art galleries were closed to the public, an idea was fostered out of the Jerusalem Botanical Gardens to host an outdoor art exhibit, but not in the usual sort of way. Rather an invisible art exhibit, at least to the naked eye, and one that was concurrently displayed in 12 gardens worldwide with the same pieces of original art. “How?”, you ask. With augment reality (AR), creating the first such multi-location, simultaneous art exhibition of its kind with a number of artists, many of whom were new to AR, creating their first pieces in augmented reality.

Some of the goals of this exhibition were to:

  • give visitors something new to enjoy in gardens at times of year when plants may not be blooming
  • complement the natural setting with the AR art, encouraging visitors to experience it in a new way
  • give people a way to enjoy art in a shared, yet outdoor setting
  • encourage visitors to the gardens to engage with the gardens and with the art
  • collaborate with other gardens worldwide
  • create an exhibition without disturbing the gardens themselves, and keeping the carbon footprint to a minimum

The art exhibition was also designed to address shared themes of “nature, environment, and sustainability, exploring the boundaries and connections between art, technology, and nature. Both bleak and hopeful, each artwork offering a unique perspective on unresolved issues, creating thought-provoking, experiential, and contemplative spaces for viewers to immerse in.”

The artwork and artists crafting this narrative included:

Despite the art exhibition taking place concurrently in a number of different gardens, the experience is different in each garden, as the works are augmenting the unique surroundings and context of each garden.

Why Augmented Reality?

Building and purchasing sculptures and other art installations is a pretty major undertaking for a botanical gardens, and a travelling exhibition can generally only visit one garden at a time. While not without a cost, by making this art exhibition simultaneously available in a number of botanical gardens, this makes the cost of such an art exhibition less prohibitive. The fact that this art exhibition is accessible through augmented reality via digital devices, makes it possible to ‘install the pieces’ temporarily in multiple gardens at the same time.

The initial launch of this AR art exhibition during the pandemic was well timed to offer an alternative outdoor way to enjoy art, outside the enclosed space of an art gallery, and further encouraged visitors to think about botanical gardens as arts spaces, as well as nature spaces.

The curators of Seeing the Invisible talk about the importance of this AR art exhibition being phygital, a blending of a digital experience with a physical one. Thought was put into where each artwork was placed in each garden to enhance both the artwork and the garden, as well as take visitor safety and the protection of the planted exhibitions into consideration. The AR artwork, through the Seeing the Invisible app, was geotagged to specific locations in the gardens they were exhibited in, and visitors could only experience the artwork in that location, and only in those gardens. Thanks to the phygital, the experience was unique in each garden.

The Augmented Reality Experience

The Seeing the Invisible artwork is experienced at the gardens hosting this AR art exhibition through visitors’ smartphones and tablets via a GPS triggered app.

Viewing AR art, Dawn Chorus, through a smartphone.

To see and experience these dynamic and engaging pieces of artwork, people visiting the gardens hosting this AR art exhibition need to:

  • Before visiting a Gardens hosting the AR art exhibition:
    • Download the Seeing the Invisible App to the smartphone or tablet with cellular capabilities that they will be using at the AR art exhibition.
    • Allow the app access to device’s camera and microphone.
    • Fully charge the device before visiting the gardens.
    • Take earbuds or headphones compatible with the device to the gardens with you.
  • At a Gardens hosting the AR art exhibition:
    • If you don’t have a smartphone or tablet, you can borrow one on site.
    • Follow the map in the app to the different pieces of AR art within the gardens.
    • Follow the Seeing the Invisible App’s instructions at each piece of AR art, scanning the ground where the piece of art has been virtually installed until the art virtually appears.

The Royal Botanical Gardens shares further tips on enjoying Seeing the Invisible Art Exhibition in their gardens, here.

Once the art virtually appears before you, visitors are encouraged to engage with it and fully immerse themselves in it, by listening through their headphones to the experience, and walking around the artwork and even into it. Visitors can also take pictures with the virtual art and read more about the art through the app.

Each piece of art, lends to different forms of engagement. For instance with:

  • Gilded Cage: You can walk into the labyrinth of cells.
  • Dawn Chorus: Birds swoop around you as you are drawn in by the music from the piano.
  • Biome Gateway: You can walk into the temple cave, and discover a portal to walk through and into a parallel landscape.
  • Forget Me Not: By moving toward the flower arrangement, you trigger a gun to propel a bullet through the flower arrangement causing it to explode outwards. Once the explosion has occurred, if you walk around the vase, you will hear three different scholars discussing the flower arrangement.
  • Pneuma: By walking into the sphere, you see the gardens around you through the distortion of being inside a bubble, while the sound of breathing acts to change your own breathing pattern while experiencing.

Morphecore Prototype AR even inspired a dancer to choreograph their own piece to dance along with. Other host Gardens similarly witnessed many of their visitors dancing along with the Morphecore Prototype.

My Reflections on this Augmented Reality Experience

One of my goals in exploring different extended reality experiences is to discover what excites me in these experiences and what I find challenging, in order to reflect on what could create even richer and more accessible extended reality experiences.

Here are my thoughts on the Seeing the Invisible AR Art Exhibition, from when I visited it in September of 2022 at the Royal Botanical Gardens.

Signs welcoming visitors to the Seeing the Invisible AR Art Exhibition at the Royal Botanical Gardens.

The Extended Reality Magic

This is a clever idea to get visitors exploring the gardens through a new lens, allowing the augmented reality art to spark their imagination in a new way, to perceive the gardens differently, and to provoke thought pertaining to nature, technology and art.

Some art, like Dawn Chorus, brought pure joy of the fantastically, magical whimsy that augmented reality brings with it, further adding beauty to the surrounding gardens.

AR birds flying around an augmented reality piano in a real world garden.
AR birds flying around an augmented reality piano, care of Dawn Chorus, in a real world garden.

Other art, like Pneuma, made you feel as though you’d stepped out of reality and were peering back in at it.

AR Art looking back at the gardens through the distortion of the augmented reality bubble created by Mel O’Callaghan's Pneuma.
Looking back at the gardens through the distortion of the augmented reality bubble created by Mel O’Callaghan’s Pneuma.

While other art, like Gilded Cage, got you questioning our problematically omnipotent and controlling relationship with nature from the perspective on an artist who was imprisoned.

AR art, Ai Weiwei's golden Gilded Cage, casting augmented reality shadows, by a park bench in a real world garden.
AR art of Ai Weiwei’s golden Gilded Cage, casting augmented reality shadows by a park bench in a real world garden.

Best of all, the gardens discovered that people of all ages wanted to engage with the art, even dancing along to Daito Manabe’s Morphecore Prototype AR.

A real world dancer at Jerusalem Botanical Gardens dancing alongside Daito Manabe's Morphecore Prototype AR.
A real world dancer at Jerusalem Botanical Gardens dancing alongside Daito Manabe’s Morphecore Prototype AR.

Logistically the GPS triggered app is a good idea to create a curated experience within each garden, and to host this simultaneously between different gardens.

Adding in the ability for visitors to be able to photograph the AR art through the app, added in the ability for visitors to have fun engaging with the art. This also added to crowdsourced storytelling around the exhibit, which I can only imagine was beneficial to marketing the exhibition.

Visitors at a number of gardens playing with Pneuma selfies.
Visitors at a number of gardens playing with Pneuma selfies.

I’d love to experience future such AR art exhibitions at the Royal Botanical Garden, as well as to experience the Seeing the Invisible Art Exhibition in different gardens to see how the experience differs between the gardens with the same art.

Current Challenges (as of 2022 Experience)

Excited to experience the Seeing the Invisible Art Exhibition at the Royal Botanical Gardens in Burlington, Ontario, my niece and I followed the instructions and downloaded the Seeing the Invisible App to our smartphones before heading to Hendrie Park to enjoy the exhibit together.

Seeing the Invisible at the Royal Botanical Garden's Hendrie Park.

The only problem, when we got there, the Seeing the Invisible App didn’t work on either of our smartphones, despite both having compatible technology. Not sure if this is the reason why, but one woman told us the issue was due to the strength of the wifi signal from your data provider.

No problem, we queued to borrow one of tablets that the gardens had available for visitors that needed it.

Borrowing a loaner tablet at the Royal Botanical Gardens.

While the borrowed tablet worked, this now meant that we were now sharing a device, our headphones didn’t work in the borrowed device, and the borrowed tablet had a screen protector over it, that made the AR more difficult to see, especially in sunny areas. Added to this, almost every piece of AR art was placed in a sunny area, and it was hot, so not only were we battling with the light to see the AR art through the screen, but needed shade for respite for the sun.

I assumed that this user experience oversight was down to this being the Royal Botanical Garden’s first AR Exhibition and them not being familiar enough with the technology to understand how to create a positive user experience, but it turns out it was actually the curators that mapped out where each piece would be placed. Maybe the oversight of the sun was made, as due to the pandemic, the curators did not visit Hendrie Park before deciding where to place the AR art, but either which way, this emphasizes the importance of taking the user experience into consideration when designing AR exhibitions. Shade is important, both for the visitors’ physical comfort and for the practicality of visitors actually being able to properly see and experience the AR art work through their screens. With the size of the Royal Botanical Gardens there is more than enough options to place these 13 pieces of artwork in a way that keeps visitors out of the full sun and allows them to better see and experience the AR art on their screens.

Girl holding a tablet, demonstrating the difficulty seeing the AR art in the sun.

While certainly the audio experience would have been much better had we been able to use our headphones, as people do experience exhibits like this on a shared device, I think it would be beneficial for the audio portion of the experience to be added at a higher volume to better allow for visitors sharing a device to be able to hear the experience.

I appreciated that the artist shared a write up on their art within the app, but to spend time reading this, it takes you away from viewing and experiencing the AR art. It would be great, if visitors have the additional option to play an audio recording of the artist sharing their thoughts with visitors on the piece of art.

Finally, with the loaned tablets, it would be beneficial if the individual gardens made it possible for visitors to email themselves the copies of any photos they shot with the AR art.

Takeaways From This Extended Reality Experience

When easily visible and heard, the Seeing the Invisible Art Exhibition creates a glimpse into a hidden world that encourages thought and exploration through a new lens and perspective. Thats exciting, and demonstrated what excites me about sharing stories with augmented reality.

However, if the app isn’t working on people’s devices or the AR is difficult to see on people’s screens, then people become frustrated, rather than excited by the experience. This is why user design and testing is so very important. This should be thought of both in terms of the technology and visitors’ interactions with the physical environment in which they are experiencing the AR art. In the case of the RBG, the gardens and not trampling plants seems to have been taken into considerations, but not visitors’ physical comfort from the sun or thought of how that sun would impact user’s experience. This art exhibition was almost in its entirety in direct sun with no to very little shade.

A girl in another full sun AR art exhibit spot at the Royal Botanical Gardens.

User testing should be inclusive of the loaner tablets for visitor use, made available onsite. This means, if adding protective screen covers to the tablets, making sure those protective screen covers do not hinder visitors’ ability to enjoy the AR art.

It would also be advantageous to allow people using the borrowed tablets to be able to email themselves any pictures they took with the AR art, or if that is not possible, to be upfront with visitors about that from the start, so as not to leave them disappointed at the end of the experience.

Summary of Takeaways:

  • The technology exists to create a GPS triggered augmented reality app that works on people’s devices without overloading a device’s data.
  • The Seeing the Invisible AR App takes up 2.1 GB of space on my smartphone.
  • The Seeing the Invisible AR App was developed by Khora ApS, a virtual reality and augmented reality production studio in Copenhagen.
  • When the AR app is working, the augmented reality art exhibition creates wonder for visitors, gets visitors exploring and interacting in a new way, and has visitors engaging with the AR art.
  • By making the AR art exhibition viewable through smartphones and tablets (with loaner tablets available for use), it is accessible to all ages and most abilities.
  • User testing is important on the site of the experience, thinking about user comfort at different times of year.
  • Avoid creating GPS triggered AR screen experiences in direct sun.
  • Make sure protective screen covers on loaner tablet do not limit the ability to see the augmented reality.
  • Think about user safety and comfort when placing GPS triggered AR experiences.
  • Encourage visitors to bring headphones, while ensuring sound is available at a volume that those without headphones or sharing a device can hear.
  • People enjoy interacting with AR art.
  • Make sure people can email themselves any pictures they took with the AR art on loaned tablets.

Future Building

In reflecting upon the Seeing the Invisible AR Art Exhibition, in addition to placing the augmented reality art in spaces that people can enjoy the art in the shade and to be better able to hear the accompanying sounds when experiencing through a shared device, I’d also love to further encourage the engagement that people enjoyed with the art. Some possible ideas for that:

  • I’ve recently seen how the Relive App makes it easy to create videos from a walk by recording the map and editing in the photos and videos you take at different stops. Should this be possible within this AR app, it could create fun keepsakes from the AR experience, that also become valuable crowdsourced storytelling for the exhibit. An example from the travels of a fellow member of Vancouver’s tech community:
  • It would be great to be able to play the artist’s description of their work as an audio reading, complete with some prompts to encourage people to engage with their art in different ways.
  • I loved how Ori Gersht’s Forget Me Not was responsive to visitor’s movements, it would be exciting to have more pieces that respond to the location of the holder of a device.
Ori Gersht's Forget Me Not AR art flower arrangement.
Exploding vase of flowers, as a part of Ori Gersht's Forget Me Not responsive AR art.

The educator in me has also begun to design, in my head, a scavenger hunt that encourages visitors to collect experiences and different perspectives with each piece of art by encouraging visitors to take on some sort of challenge or unravel some sort of mystery with each piece of art. Being able to create a mapped storytelling account of such a scavenger hunt, in a format like the Relive App creates, would be a great way to further share the discoveries from such a scavenger hunt.

The storyteller in me would also love to learn more about the making of the art, like with this talk on Ori Gersht’s and Timur Si-Qin’s AR art pieces.

What’s Your Take on this Augmented Reality Experience?

I’d love to hear your thoughts in comments below on what you think would make this a richer and more accessible AR experience, and if you have technical solutions for making this a more immersive and user friendly experience.

One visitor using an umbrella to improve his AR art experience at the Seeing the Invisible Art Exhibition at the RBG, with all the sun.
One visitor using a black umbrella to improve his AR art experience in the sun.

References

Conklin, M., [Tucson Botanical Gardens], Maor, H., & Haring, T. (2022, December 1). A Talk with the Curators of Seeing The Invisible [Video]. YouTube. https://www.youtube.com/watch?v=1BXqc0g8D_c

Horwitz, L. (2023, June 12). phygital. Customer Experience. https://www.techtarget.com/searchcustomerexperience/definition/phygital

Jerusalem Botanical Garden. (2021, November 14). אחת העבודות בתערוכת האמנות, תפתיע אתכם במיוחד. . .. Jerusalem Botanical Garden’s Facebook Page. https://www.facebook.com/Jerusalem.Botanical.Gardens/videos/372403381331281

Rendell, H., Gertler, C., Katri, M., Maor, H., & Haring, T. (2021). SEEING THE INVISIBLE. SEEING THE INVISIBLE. https://seeingtheinvisible.art/

Royal Botanic Garden Edinburgh, Maor, H., Gersht, O., & Si-Qin, T. (2022, May 24). Seeing the Invisible  In Conversation with the Artists [Video]. YouTube. https://www.youtube.com/watch?v=3_XD4nmV-58

Royal Botanical Gardens. (2023, March 8). Seeing the Invisible – Royal Botanical Gardens. https://www.rbg.ca/things-to-do/art-in-the-gardens/seeing-the-invisible

Shamir, R., [America-Israel Friendship League], Gertler, C., Rendell, H., Maor, H., Haring, T., Rominiecki, J., & Firestone, W. (2021, November 8). Seeing the Invisible: Augmented Reality art [Video]. YouTube. https://www.youtube.com/watch?v=TIuxE1oZKtY

Parco Archeologico di Tremona-Castello AR Experience – an Extended Reality Case Study

by 8 Comments

Welcome to our new series exploring extended reality case studies, in a quest to develop richer and more accessible Extended Reality (XR) experiences. In this first case study, we delve into Parco Archeologico di Tremona-Castello’s (Tremona Castello Archaeological Park) Augmented Reality (AR) Experience and speak with Nadia Lupi, one of the visionaries behind the project.

Organization: 

Parco Archeologico di Tremona-Castello (Tremona Castello Archaeological Park)

Project:

Parco Archeologico di Tremona-Castello AR Experience

Type of Experience:

Augmented Reality Walking Tour

Device for Experiencing:

Augmented Reality Glasses paired with a guided audio tour

Link to Project:

https://www.parco-archeologico.ch/en/visita/la-realta-virtuale.html

Interview / Partook in the Experience Date:

April 30, 2019

Interview With:

Nadia Lupi, Director of Mendrisiotto Turismo

Country:

Switzerland

Delving Into the Tremona Castello Archaeological Park AR Experience & an Interview with Nadia Lupi

The Project : Parco Archeologico di Tremona-Castello AR Experience

Parco Archeologico di Tremona-Castello

Sitting on top of the hillside over looking the village of Tremona in the Mendrisiotto District of the Canton of Ticino, Switzerland, local archaeologist Alfio Martinelli, discovered the archaeological remains of an ancient medieval village sharing a history of the area for 6,000 years from prehistory to the 14th century. As Alfio and other archaeologists and volunteers began to uncover this village, many questions arose as they began the puzzle together the stories from the past. Among the curiosities of Parco Archeologico di Tremona-Castello are why they chose to build there, with no apparent water source on site.

Who lived here? How did they spend their daily lives? Why did they suddenly abandon this village in the 14th century? These are all questions that Alfio, fellow archaeologists, and volunteers excavating Parco Archeologico di Tremona-Castello have been asking, as they piece together pictures and stories of daily life in this ancient village when it was thriving.

It is believed that this rural village was inhabited by artisans and farmers from dark ages until the 14th century, who travelled widely and traded their goods. The people there made widespread use of money and owned beautifully handcrafted ornamental items. The site was likely established for the village due to its strategic position, which allowed its inhabitants to control the communication routes from Como and Varese to Riva San Vitale, Monte Ceneri and the central Alps. Excavations have brought to light a labyrinthine and well-fortified settlement that offered its inhabitants refuge from Barbarian invasions and the skirmishes between Como and Milan.

Why Augmented Reality?

These are stories that Alfio and others excavating the site wanted to share, along with Mendrisiotto Turismo. Initially Alfio and Nadia Lupi, the Director of Mendrisiotto Turismo, envisioned turning Parco Archeologico di Tremona-Castello into a open air, living history museum.

They did not get approval for an open air, living history museum, however, so had to think of alternative ways of helping others to envision life in Parco Archeologico di Tremona-Castello in medieval times. Thanks to 3D Artist and Animator Elia Marcacci volunteering on the excavation of Parco Archeologico di Tremona-Castello, Alfio and Nadia got the idea to have Elia create 3D animations of life in Parco Archeologico di Tremona-Castello in medieval times, which gave them the idea to create an augmented reality experience that would allow visitors to Parco Archeologico di Tremona-Castello to envision the site in medieval times, while still seeing the present day excavation.

The Augmented Reality Experience

In creating this augmented reality experience, Alfio, Nadia and Elia worked with the University of Applied Sciences and Arts of Southern Switzerland (SUPSI) to help shape and technically guide this experience.

They wanted to create an experience that would allow visitors to envision and experience life in the ancient village, while still seeing the excavated foundations of the ancient walls on the site today.

Initially Nadia had imagined this being experienced in AR through a smartphone or a tablet, with ghostly walls of the village and ghostly figures of villagers wandering about around you, going about their daily work and life.

While they do not have that experience yet, they do have an 10 detailed 3D scenes into the past, beautifully illustrated by Elia, including places where you can visualise the ancient village all around you.

The Medieval Village that once stood at Parco Archeologico di Tremona-Castello, as depicted by Elia Marcacci.
The Medieval Village that once stood at Parco Archeologico di Tremona-Castello, as depicted by Elia Marcacci.

Bordering on Italy there was concern that if this experience were made accessible through smartphones and tablets, then visitors would run into issues with their devices accidentally switching to Italian telecommunication networks and resulting in additional charges. Thus the AR storytelling was created to be experienced through AR glasses instead.

To make this a richer experience, sharing audio storytelling about the ancient village and Parco Archeologico di Tremona-Castello, the AR glasses were paired with an audio guide that both shares stories on the 10-minute hike from the visitors centre to Parco Archeologico di Tremona-Castello and within Parco Archeologico di Tremona-Castello.

Due to the AR glasses and the audio guide, visitors can only partake in this experience when the visitor’s centre is open. You can get the address and hours of the visitor’s centre by clicking the button below, as well as details on renting the AR glasses and the audio guide.

Plan for an AR Walk through Parco Archeologico di Tremona-Castello

My Reflections on this Augmented Reality Experience

Part of my goal in exploring different extended reality experiences is to discover what excites me in these experiences and what I find challenging, in order to reflect on what could create even richer and more accessible extended reality experiences.

Here are my thoughts on Parco Archeologico di Tremona-Castello’s AR experience, from when I visited it in 2019.

The Extended Reality Magic

The concept video for Parco Archeologico di Tremona-Castello’s AR experience that Elia, Alfio and Nadia designed is beautiful and would make for an incredible experience for visitors, immersing them in two times / worlds simultaneously through ghostly echos from the past.

Elia’s 3D imagery is also well researched, well thought out, beautifully illustrated, and depicts incredible detail.

A household scene, as depicted by Elia Marcacci, in in the Medieval Village once at Parco Archeologico di Tremona-Castello.
A household scene, as depicted by Elia Marcacci, in in the Medieval Village once at Parco Archeologico di Tremona-Castello.

Based on memory, the audio storytelling was also well done, to transport visitors through time and help them to better imagine life in the ancient village at Parco Archeologico di Tremona-Castello.

Current Challenges (as of 2019 Experience)

As Nadia mentioned, the Parco Archeologico di Tremona-Castello’s AR experience is not yet as she envisioned it in their concept video. In 2019, rather than seeing an active ghostly village around you, you can visit a number of 3D graphic illustrations of life in the village that are generally static, aside from minor movement, like a wisp of smoke. While these 3D illustrations are well done, when you are expecting to walk into an active ghostly village, this is disappointing.

For me, however, the most problematic part of this AR experience is the AR glasses. They are uncomfortable, are difficult to see through, especially if you yourself wear glasses, and are difficult to see the 3D scenes through. As such, visitors don’t get to fully appreciate how well Elia’s illustrations have been crafted.

Erica wearing AR glasses at Tremona Castello Archaeological Park

While the audio storytelling is well designed, the audio storytelling device is problematic, as it has to be set in the visitor’s centre and once you start it, it cannot be paused or replayed without returning to the visitor’s centre to have it reset. This means you can’t take the time to enjoy and immerse yourself in the Parco Archeologico di Tremona-Castello’s AR experience at your own pace, but must stick to the pace of the device. This also means that on a hot, sunny day, you can’t take breaks from the tour under the shade of a tree, but must stay in the sun, until the tour reaches its completion.

The AR glasses and audio guide also limit when visitors can enjoy the experience to the hours of the Visitor’s Centre.

Takeaways From This Extended Reality Experience

Nadia’s initial description of what she envisioned for Parco Archeologico di Tremona-Castello’s AR experience is similar to what I’ve been envisioning for the natural and cultural history augmented reality experiences that I’ve been dreaming of. It is exactly the experience that I was hoping for, and I suspect that with advancements in motion capture suits and world building becoming more accessible through game design engines, like Unreal Engine and Unity, it is becoming easier to make visions like this a reality in the future, making me excited for a future visit to Parco Archeologico di Tremona-Castello to see how the AR experience has evolved. It would seem that Elia has already constructed and made it possible to explore the outdoor spaces of the ancient village that once stood at Parco Archeologico di Tremona-Castello through his 3D animations. Having said that, I think tackling this in pieces of starting with specific scenes, rather than a whole ghostly village all at once, makes this much more manageable logistically and financially, as well as allowing for the opportunity to test drive and learn from the initial scenes before designing more.

The AR glasses were very problematic for me, however, and create an accessibility challenge and I imagine a large expense in establishing this experience. I personally would design a version of this that visitors could enjoy through their smartphones and tablets. With advances in smartphone and tablet technology, the concerns for visitors’ devices switching to an Italian telecommunications network, resulting in unexpected cellular charges, could be resolved in the following ways:

  • Creating a GPS triggered app for the AR experience, that visitors download before leaving the visitor’s centre. Visitors can then turn on airplane mode on their devices, and start the app while using the visitor’s centre wifi. A number of travel audio guide apps work this way, including the popular GuideAlong audio tours.
  • Create a WebAR version of the AR experience, and install a mobile wifi router (like a starlink mobile) at Parco Archeologico di Tremona-Castello for visitors to use while their devices are in airplane mode.

Being able to enjoy this AR experience through a smartphone or tablet would make it accessible to more people, would make Elia’s 3D illustrations easier to view and enjoy, and would allow people to enjoy the experience at their own pace, pausing and replaying it, if they wish. In this way, the AR experience could even be made accessible outside of the visitor’s centre’s hours, if so desired.

It should be noted that one advantage to AR glasses over WebAR is that the augmented realities resolution is usually better with AR glasses, but if the augmented reality is difficult to see through the AR glasses, then it doesn’t really matter how good the resolution is.

Food storage in the Medieval Village once at Parco Archeologico di Tremona-Castello, as depicted by Elia Marcacci.
Food storage in the Medieval Village once at Parco Archeologico di Tremona-Castello, as depicted by Elia Marcacci.

Summary of Takeaways:

  • creating a ghostly ancient village that can be viewed with augmented reality with ghostly figures actively taking part in village life is where the magic in this experience is for me
  • creating a series of ghostly scenes from the village is a more manageable starting point, and allows for the opportunity to test drive and learn from the initial scenes before designing more
  • AR glasses are uncomfortable for some and difficult for some people to see the augmented reality scenes in
  • AR and audio storytelling is more accessible when experienced through a smartphone or a tablet
  • concerns of smartphones or tablets switching to an Italian telecommunications network, could be resolved by having the AR experience accessible through a GPS triggered app or available through WebAR if a mobile wifi router was available for use at Parco Archeologico di Tremona-Castello for visitors to use while their devices are in airplane mode
  • being able to enjoy this AR experience through a smartphones or tablet would allow people to immerse themselves in the experience at their own pace, pausing and replaying it, if they wish

Future Building

In reflecting upon Parco Archeologico di Tremona-Castello’s AR experience, in addition to the desire to have this experience accessible through a smartphone or table, allowing me to pause and replay the audio storytelling, and to see the ancient village in ghostly form with active ghostly villagers, I’d also love to be able to interact with those scenes in some way. Some possible ideas for that:

  • the ability to take pictures with the ghostly village and villagers, including selfies
  • being able to listen in on the conversations of ghostly villagers
  • being able to learn an activity from the ghostly villagers, perhaps as they teach a ghostly child or apprentice
  • being able to partake in a festive event, by dancing or signing along with the ghostly villagers

The educator in me is also envisioning creating the village in VR for use in school classrooms, or possibly VR (or AR) teaser experiences that would get school kids and other visitors prepared for a visit with a bit of a background of the time period that they will be transported to and some questions or things to look out for and think about on their visit to Parco Archeologico di Tremona-Castello.

Exploring Tremona-Castello Archaeological Park with AR glasses.

What’s Your Take on this Augmented Reality Experience?

I’d love to hear your thoughts in comments below on what you think would make this a richer and more accessible AR experience, and if you have technical solutions for making this a more immersive and user friendly experience.

References

Hargreave, E., [StoryToGo]. (2023b, December 15). Tremona Castello Archaeological Park AR Experience [Video]. YouTube. https://www.youtube.com/watch?v=6f8NZaeC_do

La storia. (n.d.). https://aram-ti.ch/il-villaggio/la-storia/

Marcacci Elia. (2014, January 6). Ein Tag in Tremona-Castello. Medienarchiv. https://medienarchiv.zhdk.ch/entries/63853132-1225-498b-ba50-c0b38d3a9eb9

Martinelli, A. (2008). Tremona Castello. Dal V millennio a.C. al XIII secolo d.C. All’Insegna del Giglio.

MendrisiottoTurismo. (2016a, May 25). Il villaggio di Tremona-Castello in 3D [Video]. YouTube. https://www.youtube.com/watch?v=4gzp_A9ADrQ

Mendrisiotto Turismo. (2016b, August 10). Parco archeologico di Tremona-Castello, oggi [Video]. YouTube. https://www.youtube.com/watch?v=ri7HMuhSrVQ

MendrisiottoTurismo. (2016c, August 29). Anno Domini – Rievocazione storica al Parco archeologio di Tremona-Castello [Video]. YouTube. https://www.youtube.com/watch?v=hkCdTiknS-4

MendrisiottoTurismo. (2019a, May 15). Parco Archeologico di Tremona Castello [Video]. YouTube. https://www.youtube.com/watch?v=qHGKHO4bZBI

MendrisiottoTurismo. (2019b, July 12). Hunting for mysteries in time [Video]. YouTube. https://www.youtube.com/watch?v=Ty9LxvWVsOw

Mendrisiotto Turismo. (2023, May 19). La Regione da scoprire – Il Parco archeologico di Tremona 2023 [Video]. YouTube. https://www.youtube.com/watch?v=6BbQJVkru30

Parco Archeologico di Tremona-Castello (n.d.). Ticino Tourism. https://www.parco-archeologico.ch/en/

Società editrice del Corriere del Ticino SA. (n.d.). Tremona – TicinoTopTen. Ticino Top Ten. https://www.ticinotopten.ch/en/experiences/tremona

Exploring XR Development with my Final Master of Educational Technology Directed Studies

by 4 Comments

Join me in exploring XR Development for my final Master of Educational Technology Directed Studies, and finding ways of making XR Development more manageable and accessible for other educators, students and creatives navigating XR and beginning to develop their own projects.

One of my main goals in embarking upon my Masters of Educational Technology was to explore newer-to-me forms of storytelling that could help to bring education to life. Over the past 7 years as I’ve worked my way through my Masters, being introduced to UBC’s Emerging Media Lab and the work they are doing there, having been approached by a board member from the Juno Beach Centre to explore newer ways of sharing stories from World War II that would connect with Canadian school children, travelling to Normandy to visit the museum and see how they are sharing stories there and identifying the gaps in their storytelling, and being shortlisted and interviewed for an interactive and immersive storytelling fellowship in Norway and exploring the stories I might wish to create for that fellowship, the idea of delving deeper into education and storytelling through XR – Extended Reality, teased at my synapses. I took an early course in VR in Education, attended XR symposiums and showcases, began researching and exploring different applications of XR in education and storytelling, collaborated on an open course in Immersive Experiences in Natural and Cultural History Education, developed an idea for an AR storytelling app to share natural and cultural history experiences, and am working my way through other people’s courses in XR, including the University of Michigan’s Extended Reality series of courses. While I am only now beginning to write about all of what I have learned here, the series of posts that follow come from research and a rabbit hole that I began to travel through 6-years-ago, and to date have included four talks at national and international conferences in education, technology, and storytelling, sharing my findings and explorations, and one talk to a Journalism Masters Class at the American University of Cairo. I have shared that last talk below as it shares many of the goals that I set out to accomplish with this directed studies.

Defining Immersive and Interactive Media

Before we get into where my interests and observations are taking me with my final directed studies for my Masters, I thought we’d explore the terminology around immersive media, experiential media, and extended reality, as there is ambiguity around the terminology, so by addressing this at the onset, I hope to at least make it clear as to how I am using the terminology.

An Immersive art exhibit entitled, Interactive Diorama - Rembrandt 1632.
Interactive Diorama—Rembrandt, 1632, The Anatomy Lesson of Dr. Nicolaes Tulp / Aalto University, School of Arts, Design and Architecture, Helsinki. Photographed by Tom Mesic.

Immersive and Interactive Media / Experiential Media

Immersive and Interactive Media are basically the current term for what was termed transmedia, referring to media that is immersive and interactive in nature, allowing those that are engaged with it to experience it in a way that allows them to step into the experience and / or become a part of the media and / or interact with the media, potentially having an impact on how the experience is shaped.

Examples of forms of immersive and interactive media include virtual reality, augmented reality, escape rooms, 360 video, choose your own adventure stories, alternate reality games … and the list goes on. Another name for this form of media is experiential media

Extended Reality (XR)

Project Lead Olivia How, demonstrating Shakespeare xR's touch table of Shakespeare’s First Folio to interested onlookers.

Extended Reality is a form of immersive / experiential media that takes users into a new reality or places virtual objects into a user’s real world, enhancing everyday life with technology. Basically making impossible sensorial experiences, possible. (Hargreave, 2021a)

Included within extended reality are virtual reality (VR), augmented reality (AR), and mixed reality (MR).

Virtual Reality (VR)

Virtual Reality is a computer-generated environment with scenes and objects that appear to be real, making the user feel they are immersed in their surroundings. (Iberdrola, n.d.) While this often is experienced through the use of a headset, a user could also walk into a virtual reality environment created by projectors and screens. This simulated environment and experience could be similar to or completely different from that of the real world in which a user is physically situated.

Examples of virtual reality include 360 video, when experienced through a headset, meeting in Mozilla Hubs, and the Immersive Van Gogh Exhibit.

A student experimenting with the Occulus Quest in the Emerging Media Lab at UBC.
A student experimenting in VR at the Emerging Media Lab at UBC.

Augmented Reality (AR)

Augmented Reality (AR) is an interactive experience where the real, physical world is digitally augmented. The augmentation is usually visual in current applications, but it can also be through other senses, such as auditory or haptic. (Wong, 2021)

Examples of augmented reality include Pokemon Go, the ReBlink exhibition at the AGO, and selfie masking filters in SnapChat.

Exploring Tremona-Castello Archaeological Park with AR glasses.
Exploring Tremona-Castello Archaeological Park with AR glasses.

Mixed Reality (MR)

There is a lot of debate around what constitutes Mixed Reality. I have always thought of it and defined it as allowing for real and virtual elements to interact with one another, and for users to interact with virtual elements in a similar way they would in the real world or at least in a similar way that they might interact with a touch screen. (Hargreave, 2021b)

Students observing the Holobrain at UBC.
Observing the Holobrain at UBC, care of Kerry Blackadar.

Others have defined mix reality (Speicher et al, 2019) as:

  • a synonym for augmented reality
  • a combination of AR and VR
  • a stronger version of AR
  • a type of collaboration
  • the idea of aligning physical virtual environments. 
  • the spectrum from the real environment without any augmentation to the virtual environment, which is completely synthesized computer-generated virtual content. 

This is further confused by some researchers only working within the theoretical and not experimenting with the tools in real time and some corporations attempting to use the term ‘mixed reality’ to market the next level advances in extended reality.

Due to all the debate around defining mixed reality, and based on one research paper, the unlikelihood of a consensus between the various experts in research and industry, I will likely avoid the term mixed reality and instead describe the characteristics of various extended reality applications, and possibly where they lie on the reality–virtuality continuum developed by Paul Milgram and colleagues in 1994.

Reality-Virtuality Continuum

The Reality-Virtuality Continuum as proposed by Paul Milgram, Haruo Takemura, Akira Utsumi, and Fumio Kishino in their 1994 paper, encompasses all possible variations and compositions of real and virtual objects.  They hypothesized that everything between real and the virtual is mixed reality, and that mixed reality makes up both augmented reality and augment virtuality. (Milgram et al, 1994) Basically suggesting that mixed reality is the blending of the physical and the virtual world with varying degrees of augmentation. 

The problem with the RV Continuum in its current form is that it only takes into account visual, not sound, olfactory or haptics senses. (Speicher et al, 2019) It ultimately needs to be redeviced with how technologies are evolving.

Augmented Virtuality

Augmented Virtuality refers to perceiving mostly virtual content, while still seeing some real world content. (Nebeling, 2022)

My Interests and Explorations in Extended Reality

Virtual Reality

I have long been fascinated with extended reality, ever since I was first introduced to the concept of virtual reality in an episode of Murder She Wrote in 1993. It excited me to imagine how extended reality could be used to create immersive story experience and bring worlds to life that are created in one’s imagination. The problem for me though was that whenever I tried it, which I did over the years whenever an opportunity present itself, was that it was too much sensory overload for me, causing mr to feel motion sick (which I now know is cybersickness) and sparked migraines. (Weech et al, 2019)

Jessica Fletcher in a VR headset as she solves a 1993 Murder She Wrote mystery.
Jessica Fletcher in a VR headset as she solves a 1993 Murder She Wrote mystery.

Persistence pays off though, as in 2018 I finally had a VR experience that did not spark cybersickness or a migraine. The experience in question was a virtual plank walk designed to help people overcome their fear of heights. (Basbasse et al, 2023) In exploring the features that made the difference in this experience for me and talking to one of the designers of the experience, I believe the quality of the filming / graphics made the main difference in this experience, as well as perhaps the experience being designed to limit your time in VR. This also got me inquiring from the designer if VR was being used in other ways in the medical world, and if perhaps it was being used with pain management. My interest? By this point I’d been living with chronic pain following two car accidents for 5 years, and was curious if perhaps there were ways this technology could be used to help people better manage their pain. This sparked me to begin explore different medical avenues and uses to VR, as I began to contemplate what might be useful to me and to the community of chronic pain and fatigue sufferers that I now belonged to, many of whom are predominantly home bound.

VR Plank Walk
Epionia Therapeutic VR
Therapeutic VR for Pain Reduction

Augmented Reality

I have always loved and been drawn to educational experiences that incorporate storytelling and encourage the imagination. This was developed in me as a parks naturalist and is part of how I approached science education, and encourage kids to think, imagine, experience and explore.

In approaching work and storytelling, post the car accidents that very much changed my life, how I work, what I can handle and in some ways who I am, I have been increasingly drawn back towards natural and cultural history in my storytelling, as nature soothes my symptoms. When I travel I also love learning and love to envelop myself in the story of a space, but thanks to my post accidents realities I cannot always handle busy museums with noise, lights, and other sensory triggers. This got me imagining, first with my fellowship interview in Norway and later as I explored the Juno Beach Centre in Normandy, if I could create mobile storytelling experiences outside of the museums in places of natural and cultural history, where people could learn and enjoy the stories of the place as they explored outdoors. In both cases, as they were looking to me to imagine newer ways of bringing such stories to life, I began to imagine this with augmented reality as experienced through our mobile devices. This made me smile, as not only did this idea hold the potential for learning and storytelling, but also some of the fun, whimsy and magic that augmented reality brings with it. This is an idea that I further began to sketch out on my travels and in my ETEC 522 Course – Ventures in Educational Technology, and that I am working towards making a reality.

My Goals with this Directed Studies into Extended Reality

I have a tendency to dream big and my dyslexic brain sees connections and expanding and overlapping storyworlds. Usually this means that I do not have the money and often all the knowledge for orchestrating all that I imagine at the get go. My dyslexia has taught me to problem solve though, and life and an imagination and brain that don’t let go of ideas have taught me patience and to play the long game, approaching my ideas in digestible pieces. Thats what I have been doing with this directed studies. I essentially started the work on it after the first year into my Masters, slowly taking advantage of research experiences as they became available, participating in XR symposiums and showcases when they present themselves, taking formal and informal courses in XR, accepting opportunities to speak on what I am working towards with XR and learning along the way, and beginning to flush out my ideas and pitch them to funding bodies.

For this directed studies, my goals are to:

  • Write up the VR case studies that are helping me to identify the strengths and challenges in VR for patient communities.
  • Write up the AR case studies that are helping me to identify the strengths and challenges in AR for natural and history education.
  • Begin to develop and further experiment with VR and AR development.
  • Sharing what I have learned to help make it more manageable and accessible for other educators, students and creatives navigating XR and begin to develop their own projects.

A Beginning

While in some ways I see this directed studies as an end to my researching and a beginning to my developing, there will always be more research to be done in seeing how others are developing and applying the XR, especially as the technology is still developing and despite its decades of history, still very much in its early days for development, application and adoption.

Development-wise, waiting on gear and difficulties with my health and post concussion syndrome, have very much slowed down my intended goals with development and where I’d like to be at this stage, in terms of developing my own XR applications in VR for patients and AR for natural and cultural history. However, as my post accidents self has learned, pushing through when my health issues becomes problematic, only making things worse and setting me back further. Meaning that I have had to be patient with myself, have had to get my health back to a stable place (during a year that posed a lot of challenges with that), and have had to address the backlog of pressing deadlines from my health going sideways before I’ve been able to engage in development. I’ve put in the work to do that, and am now in a good and healthy space to embark on development. This is much later than I would have liked, but that is not something I can change with my newer disability realities post the car accidents. As such, my initial development goals for this directed studies will be smaller for now, and I will continue working on the bigger goals, post the directed studies and will continue to write those up as I do.

Erica wearing AR glasses at Tremona Castello Archaeological Park

This also means that for the Guide to XR Development that I will be developing to make XR Development more manageable and accessible for other educators, students and creatives, this will similarly take shape in stages and over time, some coming sooner with my directed studies and some continuing over time, as I further delve into XR development and experiments of my own.

Thank you for joining me on this journey into XR Development, and being patient with me as I take the time to do this in a way that is healthy for me with my health and disability realities. Hopefully my journey as the tortoise on this road will inspire others to explore and embark on such a journey of their own, no matter at what speed and working with whatever challenges your life and situation might pose along the way.

References

El Basbasse, Y., Packheiser, J., Peterburs, J., Maymon, C., Güntürkün, O., Grimshaw, G., & Ocklenburg, S. (2023). Walk the plank! Using mobile electroencephalography to investigate emotional lateralization of immersive fear in virtual reality. Royal Society open science, 10(5), 221239. https://doi.org/10.1098/rsos.221239

Hargreave, E. (2021a, November 15). An introduction to immersive experiences in natural & cultural history education – StoryToGo Classroom. StoryToGo Classroom. https://storytogo.ca/classroom/course/immersive-experiences-in-natural-and-cultural-history/lessons/intro/

Hargreave, E. (2021b, November 15). Mixed Reality (MR) in Natural & Cultural History Education – StoryToGo Classroom. StoryToGo Classroom. https://storytogo.ca/classroom/course/immersive-experiences-in-natural-and-cultural-history/lessons/mr-in-natural-cultural-history-education/

Hargreave, E., [Stories, EdTech & Digital Media with Ahimsa Media]. (2020, November 29). Story Steppers – an AR storytelling app to the natural & cultural history of a place [Video]. YouTube. https://www.youtube.com/watch?v=2xeWjBWYcAI

Hargreave, E., [StoryToGo]. (2023, December 10). Exploring Immersive Media to Develop Creative & Engaging Stories with Master of Journalism Students [Video]. YouTube. https://www.youtube.com/watch?v=Ff1cMMiH5cc

Iberdrola. (n.d.). Virtual Reality: another world within sight. Retrieved December 7, 2023, from https://www.iberdrola.com/innovation/virtual-reality

P. Milgram, H. Takemura, A. Utsumi, F. Kishino: Augmented Reality: A class of displays on the reality-virtuality continuum. In Proceedings of Telemanipulator and Telepresence Technologies, 1994.

Nebeling, M. (2022, June 2). Intro to AR/VR/MR/XR: Technologies, applications & issues. Coursera. https://www.coursera.org/learn/intro-augmented-virtual-mixed-extended-reality-technologies-applications-issues

M. Speicher, B.D. Hall, M. Nebeling: What is Mixed Reality? In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI 2019, Glasgow, Scotland, UK, May 4-9, 2019.

Weech, S., Kenny, S., & Barnett-Cowan, M. (2019). Presence and Cybersickness in Virtual Reality Are Negatively Related: A Review. Frontiers in psychology, 10, 158. https://doi.org/10.3389/fpsyg.2019.00158

Wong, Y. (2021, November 15). Augmented Reality (AR) in Natural & Cultural History Education – StoryToGo Classroom. StoryToGo Classroom. https://storytogo.ca/classroom/course/immersive-experiences-in-natural-and-cultural-history/lessons/ar/

Learning and Creating with Read&Write

by Leave a Comment

In this first article of our new Inclusive Design series, we explore learning and creating with Read&Write, a literacy support tool that offers help with everyday digital tasks, including reading digital text aloud, understanding unfamiliar words, researching assignments and proofing written work. 

Tune into the video below for a walk through of the Read&Write Chrome Extension, including it’s features, my experiences with it, and this writer / educator’s thoughts on some of the ways that Read&Write can help individuals to gain understanding from the digital content they read, engage with the digital content that they read, and express themselves through their own digital writing.

Read&Write Tools / Features

Designed initially to help students to understand, engage, and express themselves with their digital reading and writing, Read&Write is just as useful for adults and youth, in educational and work settings, as well as for aiding individuals in achieving some of their personal goals.

Help students understand, engage and express themselves with Read&Write. Read&Write is a literacy support tool that offers help with everyday tasks like reading text out loud, understanding unfamiliar words, researching assignments and proofing written work.

Below are some of the reading, comprehension, and writing tools and features that Read&Write offers.

Reading Tools

These are a few of the tools that Read&Write offers to support users with accessing digital readings.

Text to Speech Reader

The Read&Write toolbar acts as a Text to Speech Reader, allowing you to listen to the text on your screen by pressing the play button, as well as pausing and stopping the reading. To help you in following along with what is being read, the sentence is highlighted in yellow and the word in blue.

Read&Write Text to Speech Reader in action.

The voice doing the reading and speed of the reading can be changed within the settings.

Audio Maker

By highlighting text and clicking on Audio Maker, Read&Write will create a Text to Speech audio reading of up to 10,000 characters of text, allowing you to listen to digital readings while offline and on the go.

Screen Masking

Clicking on Screen Masking will add a dark filter on your screen, with a light strip that moves with your cursor, to narrow your focus to the specific sentence or paragraph that you are reading.

Read&Write's Screen Masking in action.

Simplify Page

This tool creates a simplified page of the page that you are reading, free of images, videos, and ads. It also allows you to change the background and text colour on the page, the font, the size of the text, and the line spacing. In addition, it offers a Discover feature, which identifies keywords that readers might need help comprehending. By clicking these keywords, it provides a picture and a description of the what the word(s) refer to / mean.

An example of Read&Write's Simplify Page in action.

Comprehension Tools

Read&Write offers a number of tools to aid in research and reading comprehension, allowing readers to engage with the text that they are readings.

Dictionary

By highlighting a word and clicking on the dictionary, a dictionary description of all the uses and meanings of that word will appear in both audio and text form.

Read&Write Dictionary and Picture Dictionary response to 'mammoth'.

Picture Dictionary

By highlighting a word and clicking on the picture dictionary, a picture depicting that word will appear.

Web Search

By highlighting a word or phrase and clicking on the web search tool, a new tab will open taking you to a google search of that term.

Google search for 'mammoth'.

Translator

To see and listen to a word in a different language, in the settings, select the desired language for translations under ‘Speech’, and then click translate.

Read&Write's Translator tool sharing the word 'mammoth' in Arabic.

Highlighters

The Read&Write toolbar allows you to highlight text in a digital document in up to four colours and to clear those highlights at the click of a button.

Collect Highlights

By clicking the Collect Highlights tool, you can select which colour of highlights you would like collected, and it will collect those in a Google Document and include a link to the article that the highlights were collected from.

Read&Write's Collect Highlights in action.
Vocabulary List

By selecting the Vocabulary List tool, Read&Write will create a table in a Google Document with the highlighted words, their meaning, a symbol that represents them, and a column for you to add your notes.

An example of a vocabulary list created by Read&Write.

Writing Tools

Read&Write offers a number of tools to aid in the writing process.

Prediction

With the Prediction tool activated in the Read&Write toolbar while you are typing, it will offer a number of options for predictive text, helping writers with their spelling and tenses.

Read&Write Prediction tool in action.

Talk & Type

The Read&Write Talk & Type tool will type out the words you speak, when turned on, acting as a dictaphone.

Practice Reading Out Loud

By using the Practice Reading Out Loud tool, writers can read and record the text that they have written, and then listen to what they have read, in order to proofread their writing and catch any mistakes.

Uses of Read&Write for Inclusion and Accessibility

Read&Write offers tools that can aid everyone with their reading, reading comprehension, research, and writing, while at the same time offering solutions and workarounds for those that face difficulties and challenges with reading, reading comprehension, research, and writing.

Digital literacy accessibility with Read&Write.

Reading

For newer readers, individuals that struggle with reading, and those reading in a different language, Read&Write allows individuals to read-a-long with the text to speech reader, following the words being read aloud, being able to play, pause, and stop readings as they wish.

For those that learn better from auditory learning, that learn better while being active, who need to multi-task due to time constraints or need to limit their screen time, Audio Maker offers the ability to prepare audio files to listen to without the need of looking at a screen.

Read&Write's Audio Maker

For those that need to limit distractions while reading, Screen Masking, Text to Speech highlighting, and Simplify Page, all offer options for this.

For those that need the ability to manipulate their reading environment, Simplify Page offers this ability, allowing individuals to change text size, fonts, line spacing, and colour patterns to suit their individual needs and preferences.

Reading Comprehension

Whether learning new words, studying a new topic, or learning a new language, Read&Write offers readers the ability to build upon their vocabulary and understanding with ease. Read&Write allows readers to look up the meanings and uses of words in auditory, written, and pictorial form within the document they are reading at the click of a button with the Dictionary, Picture Dictionary, and Discover (within Simplify Page) tools. Readers can even build full vocabulary lists, curated to specific topics, using their colour coded Highlighters and the Vocabulary List tool.

For readers, reading in a new language, they can also use the Translator tool for the translation of words in audio and written format.

Research

Read&Write can aid in research by allowing researchers to highlight and colour code key pieces of information in their reading, and then curate and collect those highlights in a Google Doc with a link to the source using the Collect Highlights tool. Should the researcher wish more information on a particular topic, they can also use the Web Search tool to search terms with ease beyond the article.

Writing

Text Help character with a pencil.

Whether struggling with writer’s block or spelling, writing in a different language or simply needing a writing prompt, Read&Write’s Prediction tool can help with that.

And provided a writer is able to organise their thoughts in advance, Talk & Type can write up someone’s spoken words, aiding those that have challenges with typing or are in need of ways of limiting their screen time. It is also helpful for those unsure of how to spell a particular word.

Finally using the Practice Reading Out Loud tool after you are done writing, is highly useful in editing your writing, catching editing errors and disconnects in the flow of your writing.

Access Read&Write

Currently Read&Write is available for:

If you are affiliate with an educational institution, check with your institution as many have a license that allows for students and staff to use Read&Write at little to no cost. Additionally, Read&Write offers free individual plans to teachers.

Share Your Experiences with Read&Write

Have you used Read&Write? We’d love to hear of your experience with it. How do you use it? What do you find it useful for? Do you have any tips and tricks to using Read&Write?

Toolbelt Theory for Learning and Accessibility

by Leave a Comment

A trap that so many of us have a tendency to fall into is to try and help others by pushing our ideals and what works for us onto them. This can be particularly true in how we approach children, the elderly, individuals with disabilities, and those who are sick or injured; perceiving them as more vulnerable and in need of our help in making decisions for them. We forget that we are all individuals and that what works for us, may not work for them. Ultimately the only expert in what works for them is them (not you or whatever degrees and credentials you might have).

To listen to this article, click the red play arrow on the SoundCloud recording below:

Toolbelt Theory

Take for example, 3 carpentry students that are given a toolbelt. They might all start with the same toolbelt, but as they learn, experiment, and discover what works for them, it is not going to be long before each of those toolbelts are unique to each individual carpenter. This is the premise behind Ira David Socal’s Toolbelt Theory. While tools might be useful to us in performing certain tasks, how we utilise those tools and which are useful to us in the environments in which we work is very individual. Ultimately, in order to discover which tools will be useful to any one person, that person needs to know what is available to them and to have the opportunity to test the tools out in their environment, so that they can build a toolbelt that is useful to them.

Chico's well stocked tool belt from Chris Monroe's book Monkey with a Tool Belt
Chico certainly has a tool belt unique to him. Read on in Chris Monroe’s book Monkey with a Tool Belt.

Toolbelt Theory is based around the idea that as humans we are tool users, and that we choose tools most effectively when that choice begins with the Task at hand, and then considers the Environment in which that task must be performed, the Skill set of the individual (the tool chooser), and the Tools which are available. (Thus the acronym TEST).

~ Ira David Socal

Exploring Toolbelt Theory From the Perspective of Pain Management

As someone who has spent the last 9-years learning to manage pain and 5-years learning to manage post concussion syndrome, this makes a tremendous amount of sense to me. Key to what I have learned from the medical specialists, group pain management classes, and from my fellow patients is that what works for one person in their pain management, does not necessarily work for all. In teaching pain management to patients, medical specialists are essentially presenting patients with a number of tools (although in this case, we are not necessarily discussing physical tools) for managing their pain and asking the patient to experiment with the different pain management tools to figure out which methods of pain management work for them. In essence, each of us patients are creating a pain management toolbelt.

Avoiding the Pitfall of the Misguided Notion that the Educator Knows Best

The key here is allowing the individual to figure this out for themselves, which for whatever reason can be a hard for some educators to understand, perhaps due to the misguided notion that by limiting options, we are keeping things simpler for the student or saving the student time. I just need to look to my Masters to see examples of this. In asking the Accessibility Centre for help in finding ways of managing the online course readings without aggravating my concussion symptoms, I was presented with one solution – Read and Write Gold. Only after that did not work for me, was I presented with a second option on having real voice audio recordings made of my course readings. The latter helped but only with the course readings, for everything else I was left proceeding with extreme caution, walking a fine line with each course in not aggravating my symptoms too badly from all the online reading. And this leaves me with no solution for consuming online readings and research after I’m done my Masters. Fortunately in taking GDSE 584 on Universal Design for Learning and Inclusive Design and exploring the UDL Resource site, I am discovering a number of tools to navigate screen based devices to learn, research, consume content, and create content in a way that will minimise the aggravation of my concussion symptoms. Basically, instead of being given one bandaid of the brand and shape that someone else thought would work for me, I am being welcomed into a whole holistic clinic in which I am being introduced to the tools that are available, shown how to use them, and being left to experiment with them, in order for me to find which tools work best for me and in what way. Discovering these tools and recognising the possibilities that they hold has left me with such a feeling of relieve that I’ve actually cried.

Crafting My Toolbelt for Digital Learning and Creating

As I begin to create my own toolbelt for digital learning and creating, I’ve been reflected on who I am, how I learn, what my challenges are, what my strengths are, and where my passions and interests lie.

Personal Reflections on my Inclusive Design Action Plan Brainstorming Mind Map

To give you, dear reader, a bit of an overview of me as a learner, I am dyslexic, and manage chronic pain, chronic fatigue, and post concussion syndrome. I am also highly inquisitive, creative, a problem solver and an experimenter. I love storytelling, am discovering a new passion for audio storytelling, and am a biologist and educator by training. My main love in science is steeped in nature. Time in nature and with my cat are the two greatest calming influences in my life.

My current needs include:

  • mind mapping to organise my thoughts
  • tactile ways of thinking and organising
  • calming visuals that aren’t too loud
  • calming audio
  • natural voice readings
  • reminders to limit work and screen time
  • regular offline / calming breaks
  • healthy relationships
  • dimmed lighting
  • an environment in which I can control my work position, lighting and sound
  • audio recordings for remembering details from interviews
  • time in nature
  • cat snuggles

I am still fully immersed in TESTing (Task – Environment – Skills – Tools) tools with different tasks, but thus far, these are some of the tools that I am finding to be useful for me for my toolbelt for digital learning and creating:

Next up on my TESTing schedule for my toolbelt for digital learning and creating are:

In highlighting the individual nature of one’s toolbelt even when you have individuals with the same challenges, a colleague in my Masters also has post concussion syndrome and in asking him how he manages the online readings, he uses text to speech readers, which typically for me aggravate my symptoms. In addition, he also follows strict meditation practices, which is something that I find does help me.

Aiding Students in Building their Toolbelt for Learning

If toolbelts for learning are so individualistic, how do we as educators or parents help our students to build their toolbelts for learning, without falling into the trap of simply providing them with what we think they need?

What's in your toolbelt? What will help you reach your goals?
Image created with the aid of Vector Portal.

In contemplating this, this again has me reflecting back on my experiences with learning to manage my chronic pain, chronic fatigue, and post concussion syndrome in which the ChangePain Clinic, Complex Chronic Diseases Program, and Brain Injury Society of Toronto offered me different classes in learning to manage my conditions. While the classes introduced me to the tools and how they could be used, the homework in between the classes gave me the opportunity to test out tools in my own environment to see which worked for me and in what way. This is essentially what the TEST is in Toolbelt Theory.

To aid your students in creating their Toolbelts for Learning, teach them to TEST. Based on the Task they need to do, in the Environment that they need to do that task, with the Skills they have, have them try out the Tools that are available, so that they can determine which tools are the best fit for them and in which way.

Resources for Creating Toolbelts for Digital Learning and Creating

For many educators part of the challenge in aiding students in creating their toolbelt for learning, rather than simply telling students what tools / solutions they should be using, is taking the time to find what is available for their students to meet their various needs. Fortunately, Dr. Paul Hamilton has created the UDL Resource site to help with that.

A toolbelt full of digital tools for learning, creating, and exploring.
A Digital Toolbelt of Gwyneth Jones, the Daring Librarian.

One of our goals over the next few months with our StoryToGo Community Resources is to build a Library of Accessibility Resources and an open, self paced course in the StoryToGo Classroom on Building Your Digital Accessibility Toolbelt. In addition to this, we aim to create a video and blog series on StoryToGo giving people an overview of different accessibility tools and how to use them.

If you have resource suggestions for the series, library, and / or course, we’d love to hear about them in the comments. We’d also love any advice you might have on helping students to develop their toolbelt for learning.

Thanks so much!

~ Erica

StoryToGo