StoryToGo

Exploring storytelling in its many forms in this digital age

AR

Seeing the Invisible Augmented Reality Art Exhibit – an Extended Reality Case Study

by 4 Comments

In this second case study in our series exploring extended reality (XR), we examine the Seeing the Invisible Augmented Reality (AR) Art Exhibit, as exhibited by the Royal Botanical Gardens. This AR art exhibition featuring the art of 13 artists from around the world was initiated by the Jerusalem Botanical Gardens and displayed concurrently at 12 Botanical Gardens worldwide.

Project:

Seeing the Invisible Augmented Reality Art Exhibit

Initiated By:

Jerusalem Botanical Gardens in partnership with Outset Contemporary Art Fund, with the support of the Jerusalem Foundation

Curated By:

Hadas Maor and Tal Michael Haring

Host At:

Royal Botanical Gardens (RBG)

Type of Experience:

Augmented Reality (AR) Outdoor Art Exhibition

Device for Experiencing:

Smartphone or Tablet, preferably with headphones

Link to Project:

https://seeingtheinvisible.art/

Date Partook in the Experience:

September 18, 2022

Country of Origin:

Israel

Country Where I Experienced:

Canada

Reflections On My Experience

Interview Between Tucson Botanical Gardens Executive Director Michelle Conklin & Seeing The Invisible Curators Hadas Maor & Tal Michael Haring

The Project:

In 2021, as the world continued to face the pandemic and many art galleries were closed to the public, an idea was fostered out of the Jerusalem Botanical Gardens to host an outdoor art exhibit, but not in the usual sort of way. Rather an invisible art exhibit, at least to the naked eye, and one that was concurrently displayed in 12 gardens worldwide with the same pieces of original art. “How?”, you ask. With augment reality (AR), creating the first such multi-location, simultaneous art exhibition of its kind with a number of artists, many of whom were new to AR, creating their first pieces in augmented reality.

Some of the goals of this exhibition were to:

  • give visitors something new to enjoy in gardens at times of year when plants may not be blooming
  • complement the natural setting with the AR art, encouraging visitors to experience it in a new way
  • give people a way to enjoy art in a shared, yet outdoor setting
  • encourage visitors to the gardens to engage with the gardens and with the art
  • collaborate with other gardens worldwide
  • create an exhibition without disturbing the gardens themselves, and keeping the carbon footprint to a minimum

The art exhibition was also designed to address shared themes of “nature, environment, and sustainability, exploring the boundaries and connections between art, technology, and nature. Both bleak and hopeful, each artwork offering a unique perspective on unresolved issues, creating thought-provoking, experiential, and contemplative spaces for viewers to immerse in.”

The artwork and artists crafting this narrative included:

Despite the art exhibition taking place concurrently in a number of different gardens, the experience is different in each garden, as the works are augmenting the unique surroundings and context of each garden.

Why Augmented Reality?

Building and purchasing sculptures and other art installations is a pretty major undertaking for a botanical gardens, and a travelling exhibition can generally only visit one garden at a time. While not without a cost, by making this art exhibition simultaneously available in a number of botanical gardens, this makes the cost of such an art exhibition less prohibitive. The fact that this art exhibition is accessible through augmented reality via digital devices, makes it possible to ‘install the pieces’ temporarily in multiple gardens at the same time.

The initial launch of this AR art exhibition during the pandemic was well timed to offer an alternative outdoor way to enjoy art, outside the enclosed space of an art gallery, and further encouraged visitors to think about botanical gardens as arts spaces, as well as nature spaces.

The curators of Seeing the Invisible talk about the importance of this AR art exhibition being phygital, a blending of a digital experience with a physical one. Thought was put into where each artwork was placed in each garden to enhance both the artwork and the garden, as well as take visitor safety and the protection of the planted exhibitions into consideration. The AR artwork, through the Seeing the Invisible app, was geotagged to specific locations in the gardens they were exhibited in, and visitors could only experience the artwork in that location, and only in those gardens. Thanks to the phygital, the experience was unique in each garden.

The Augmented Reality Experience

The Seeing the Invisible artwork is experienced at the gardens hosting this AR art exhibition through visitors’ smartphones and tablets via a GPS triggered app.

Viewing AR art, Dawn Chorus, through a smartphone.

To see and experience these dynamic and engaging pieces of artwork, people visiting the gardens hosting this AR art exhibition need to:

  • Before visiting a Gardens hosting the AR art exhibition:
    • Download the Seeing the Invisible App to the smartphone or tablet with cellular capabilities that they will be using at the AR art exhibition.
    • Allow the app access to device’s camera and microphone.
    • Fully charge the device before visiting the gardens.
    • Take earbuds or headphones compatible with the device to the gardens with you.
  • At a Gardens hosting the AR art exhibition:
    • If you don’t have a smartphone or tablet, you can borrow one on site.
    • Follow the map in the app to the different pieces of AR art within the gardens.
    • Follow the Seeing the Invisible App’s instructions at each piece of AR art, scanning the ground where the piece of art has been virtually installed until the art virtually appears.

The Royal Botanical Gardens shares further tips on enjoying Seeing the Invisible Art Exhibition in their gardens, here.

Once the art virtually appears before you, visitors are encouraged to engage with it and fully immerse themselves in it, by listening through their headphones to the experience, and walking around the artwork and even into it. Visitors can also take pictures with the virtual art and read more about the art through the app.

Each piece of art, lends to different forms of engagement. For instance with:

  • Gilded Cage: You can walk into the labyrinth of cells.
  • Dawn Chorus: Birds swoop around you as you are drawn in by the music from the piano.
  • Biome Gateway: You can walk into the temple cave, and discover a portal to walk through and into a parallel landscape.
  • Forget Me Not: By moving toward the flower arrangement, you trigger a gun to propel a bullet through the flower arrangement causing it to explode outwards. Once the explosion has occurred, if you walk around the vase, you will hear three different scholars discussing the flower arrangement.
  • Pneuma: By walking into the sphere, you see the gardens around you through the distortion of being inside a bubble, while the sound of breathing acts to change your own breathing pattern while experiencing.

Morphecore Prototype AR even inspired a dancer to choreograph their own piece to dance along with. Other host Gardens similarly witnessed many of their visitors dancing along with the Morphecore Prototype.

My Reflections on this Augmented Reality Experience

One of my goals in exploring different extended reality experiences is to discover what excites me in these experiences and what I find challenging, in order to reflect on what could create even richer and more accessible extended reality experiences.

Here are my thoughts on the Seeing the Invisible AR Art Exhibition, from when I visited it in September of 2022 at the Royal Botanical Gardens.

Signs welcoming visitors to the Seeing the Invisible AR Art Exhibition at the Royal Botanical Gardens.

The Extended Reality Magic

This is a clever idea to get visitors exploring the gardens through a new lens, allowing the augmented reality art to spark their imagination in a new way, to perceive the gardens differently, and to provoke thought pertaining to nature, technology and art.

Some art, like Dawn Chorus, brought pure joy of the fantastically, magical whimsy that augmented reality brings with it, further adding beauty to the surrounding gardens.

AR birds flying around an augmented reality piano in a real world garden.
AR birds flying around an augmented reality piano, care of Dawn Chorus, in a real world garden.

Other art, like Pneuma, made you feel as though you’d stepped out of reality and were peering back in at it.

AR Art looking back at the gardens through the distortion of the augmented reality bubble created by Mel O’Callaghan's Pneuma.
Looking back at the gardens through the distortion of the augmented reality bubble created by Mel O’Callaghan’s Pneuma.

While other art, like Gilded Cage, got you questioning our problematically omnipotent and controlling relationship with nature from the perspective on an artist who was imprisoned.

AR art, Ai Weiwei's golden Gilded Cage, casting augmented reality shadows, by a park bench in a real world garden.
AR art of Ai Weiwei’s golden Gilded Cage, casting augmented reality shadows by a park bench in a real world garden.

Best of all, the gardens discovered that people of all ages wanted to engage with the art, even dancing along to Daito Manabe’s Morphecore Prototype AR.

A real world dancer at Jerusalem Botanical Gardens dancing alongside Daito Manabe's Morphecore Prototype AR.
A real world dancer at Jerusalem Botanical Gardens dancing alongside Daito Manabe’s Morphecore Prototype AR.

Logistically the GPS triggered app is a good idea to create a curated experience within each garden, and to host this simultaneously between different gardens.

Adding in the ability for visitors to be able to photograph the AR art through the app, added in the ability for visitors to have fun engaging with the art. This also added to crowdsourced storytelling around the exhibit, which I can only imagine was beneficial to marketing the exhibition.

Visitors at a number of gardens playing with Pneuma selfies.
Visitors at a number of gardens playing with Pneuma selfies.

I’d love to experience future such AR art exhibitions at the Royal Botanical Garden, as well as to experience the Seeing the Invisible Art Exhibition in different gardens to see how the experience differs between the gardens with the same art.

Current Challenges (as of 2022 Experience)

Excited to experience the Seeing the Invisible Art Exhibition at the Royal Botanical Gardens in Burlington, Ontario, my niece and I followed the instructions and downloaded the Seeing the Invisible App to our smartphones before heading to Hendrie Park to enjoy the exhibit together.

Seeing the Invisible at the Royal Botanical Garden's Hendrie Park.

The only problem, when we got there, the Seeing the Invisible App didn’t work on either of our smartphones, despite both having compatible technology. Not sure if this is the reason why, but one woman told us the issue was due to the strength of the wifi signal from your data provider.

No problem, we queued to borrow one of tablets that the gardens had available for visitors that needed it.

Borrowing a loaner tablet at the Royal Botanical Gardens.

While the borrowed tablet worked, this now meant that we were now sharing a device, our headphones didn’t work in the borrowed device, and the borrowed tablet had a screen protector over it, that made the AR more difficult to see, especially in sunny areas. Added to this, almost every piece of AR art was placed in a sunny area, and it was hot, so not only were we battling with the light to see the AR art through the screen, but needed shade for respite for the sun.

I assumed that this user experience oversight was down to this being the Royal Botanical Garden’s first AR Exhibition and them not being familiar enough with the technology to understand how to create a positive user experience, but it turns out it was actually the curators that mapped out where each piece would be placed. Maybe the oversight of the sun was made, as due to the pandemic, the curators did not visit Hendrie Park before deciding where to place the AR art, but either which way, this emphasizes the importance of taking the user experience into consideration when designing AR exhibitions. Shade is important, both for the visitors’ physical comfort and for the practicality of visitors actually being able to properly see and experience the AR art work through their screens. With the size of the Royal Botanical Gardens there is more than enough options to place these 13 pieces of artwork in a way that keeps visitors out of the full sun and allows them to better see and experience the AR art on their screens.

Girl holding a tablet, demonstrating the difficulty seeing the AR art in the sun.

While certainly the audio experience would have been much better had we been able to use our headphones, as people do experience exhibits like this on a shared device, I think it would be beneficial for the audio portion of the experience to be added at a higher volume to better allow for visitors sharing a device to be able to hear the experience.

I appreciated that the artist shared a write up on their art within the app, but to spend time reading this, it takes you away from viewing and experiencing the AR art. It would be great, if visitors have the additional option to play an audio recording of the artist sharing their thoughts with visitors on the piece of art.

Finally, with the loaned tablets, it would be beneficial if the individual gardens made it possible for visitors to email themselves the copies of any photos they shot with the AR art.

Takeaways From This Extended Reality Experience

When easily visible and heard, the Seeing the Invisible Art Exhibition creates a glimpse into a hidden world that encourages thought and exploration through a new lens and perspective. Thats exciting, and demonstrated what excites me about sharing stories with augmented reality.

However, if the app isn’t working on people’s devices or the AR is difficult to see on people’s screens, then people become frustrated, rather than excited by the experience. This is why user design and testing is so very important. This should be thought of both in terms of the technology and visitors’ interactions with the physical environment in which they are experiencing the AR art. In the case of the RBG, the gardens and not trampling plants seems to have been taken into considerations, but not visitors’ physical comfort from the sun or thought of how that sun would impact user’s experience. This art exhibition was almost in its entirety in direct sun with no to very little shade.

A girl in another full sun AR art exhibit spot at the Royal Botanical Gardens.

User testing should be inclusive of the loaner tablets for visitor use, made available onsite. This means, if adding protective screen covers to the tablets, making sure those protective screen covers do not hinder visitors’ ability to enjoy the AR art.

It would also be advantageous to allow people using the borrowed tablets to be able to email themselves any pictures they took with the AR art, or if that is not possible, to be upfront with visitors about that from the start, so as not to leave them disappointed at the end of the experience.

Summary of Takeaways:

  • The technology exists to create a GPS triggered augmented reality app that works on people’s devices without overloading a device’s data.
  • The Seeing the Invisible AR App takes up 2.1 GB of space on my smartphone.
  • The Seeing the Invisible AR App was developed by Khora ApS, a virtual reality and augmented reality production studio in Copenhagen.
  • When the AR app is working, the augmented reality art exhibition creates wonder for visitors, gets visitors exploring and interacting in a new way, and has visitors engaging with the AR art.
  • By making the AR art exhibition viewable through smartphones and tablets (with loaner tablets available for use), it is accessible to all ages and most abilities.
  • User testing is important on the site of the experience, thinking about user comfort at different times of year.
  • Avoid creating GPS triggered AR screen experiences in direct sun.
  • Make sure protective screen covers on loaner tablet do not limit the ability to see the augmented reality.
  • Think about user safety and comfort when placing GPS triggered AR experiences.
  • Encourage visitors to bring headphones, while ensuring sound is available at a volume that those without headphones or sharing a device can hear.
  • People enjoy interacting with AR art.
  • Make sure people can email themselves any pictures they took with the AR art on loaned tablets.

Future Building

In reflecting upon the Seeing the Invisible AR Art Exhibition, in addition to placing the augmented reality art in spaces that people can enjoy the art in the shade and to be better able to hear the accompanying sounds when experiencing through a shared device, I’d also love to further encourage the engagement that people enjoyed with the art. Some possible ideas for that:

  • I’ve recently seen how the Relive App makes it easy to create videos from a walk by recording the map and editing in the photos and videos you take at different stops. Should this be possible within this AR app, it could create fun keepsakes from the AR experience, that also become valuable crowdsourced storytelling for the exhibit. An example from the travels of a fellow member of Vancouver’s tech community:
  • It would be great to be able to play the artist’s description of their work as an audio reading, complete with some prompts to encourage people to engage with their art in different ways.
  • I loved how Ori Gersht’s Forget Me Not was responsive to visitor’s movements, it would be exciting to have more pieces that respond to the location of the holder of a device.
Ori Gersht's Forget Me Not AR art flower arrangement.
Exploding vase of flowers, as a part of Ori Gersht's Forget Me Not responsive AR art.

The educator in me has also begun to design, in my head, a scavenger hunt that encourages visitors to collect experiences and different perspectives with each piece of art by encouraging visitors to take on some sort of challenge or unravel some sort of mystery with each piece of art. Being able to create a mapped storytelling account of such a scavenger hunt, in a format like the Relive App creates, would be a great way to further share the discoveries from such a scavenger hunt.

The storyteller in me would also love to learn more about the making of the art, like with this talk on Ori Gersht’s and Timur Si-Qin’s AR art pieces.

What’s Your Take on this Augmented Reality Experience?

I’d love to hear your thoughts in comments below on what you think would make this a richer and more accessible AR experience, and if you have technical solutions for making this a more immersive and user friendly experience.

One visitor using an umbrella to improve his AR art experience at the Seeing the Invisible Art Exhibition at the RBG, with all the sun.
One visitor using a black umbrella to improve his AR art experience in the sun.

References

Conklin, M., [Tucson Botanical Gardens], Maor, H., & Haring, T. (2022, December 1). A Talk with the Curators of Seeing The Invisible [Video]. YouTube. https://www.youtube.com/watch?v=1BXqc0g8D_c

Horwitz, L. (2023, June 12). phygital. Customer Experience. https://www.techtarget.com/searchcustomerexperience/definition/phygital

Jerusalem Botanical Garden. (2021, November 14). אחת העבודות בתערוכת האמנות, תפתיע אתכם במיוחד. . .. Jerusalem Botanical Garden’s Facebook Page. https://www.facebook.com/Jerusalem.Botanical.Gardens/videos/372403381331281

Rendell, H., Gertler, C., Katri, M., Maor, H., & Haring, T. (2021). SEEING THE INVISIBLE. SEEING THE INVISIBLE. https://seeingtheinvisible.art/

Royal Botanic Garden Edinburgh, Maor, H., Gersht, O., & Si-Qin, T. (2022, May 24). Seeing the Invisible  In Conversation with the Artists [Video]. YouTube. https://www.youtube.com/watch?v=3_XD4nmV-58

Royal Botanical Gardens. (2023, March 8). Seeing the Invisible – Royal Botanical Gardens. https://www.rbg.ca/things-to-do/art-in-the-gardens/seeing-the-invisible

Shamir, R., [America-Israel Friendship League], Gertler, C., Rendell, H., Maor, H., Haring, T., Rominiecki, J., & Firestone, W. (2021, November 8). Seeing the Invisible: Augmented Reality art [Video]. YouTube. https://www.youtube.com/watch?v=TIuxE1oZKtY

Parco Archeologico di Tremona-Castello AR Experience – an Extended Reality Case Study

by 8 Comments

Welcome to our new series exploring extended reality case studies, in a quest to develop richer and more accessible Extended Reality (XR) experiences. In this first case study, we delve into Parco Archeologico di Tremona-Castello’s (Tremona Castello Archaeological Park) Augmented Reality (AR) Experience and speak with Nadia Lupi, one of the visionaries behind the project.

Organization: 

Parco Archeologico di Tremona-Castello (Tremona Castello Archaeological Park)

Project:

Parco Archeologico di Tremona-Castello AR Experience

Type of Experience:

Augmented Reality Walking Tour

Device for Experiencing:

Augmented Reality Glasses paired with a guided audio tour

Link to Project:

https://www.parco-archeologico.ch/en/visita/la-realta-virtuale.html

Interview / Partook in the Experience Date:

April 30, 2019

Interview With:

Nadia Lupi, Director of Mendrisiotto Turismo

Country:

Switzerland

Delving Into the Tremona Castello Archaeological Park AR Experience & an Interview with Nadia Lupi

The Project : Parco Archeologico di Tremona-Castello AR Experience

Parco Archeologico di Tremona-Castello

Sitting on top of the hillside over looking the village of Tremona in the Mendrisiotto District of the Canton of Ticino, Switzerland, local archaeologist Alfio Martinelli, discovered the archaeological remains of an ancient medieval village sharing a history of the area for 6,000 years from prehistory to the 14th century. As Alfio and other archaeologists and volunteers began to uncover this village, many questions arose as they began the puzzle together the stories from the past. Among the curiosities of Parco Archeologico di Tremona-Castello are why they chose to build there, with no apparent water source on site.

Who lived here? How did they spend their daily lives? Why did they suddenly abandon this village in the 14th century? These are all questions that Alfio, fellow archaeologists, and volunteers excavating Parco Archeologico di Tremona-Castello have been asking, as they piece together pictures and stories of daily life in this ancient village when it was thriving.

It is believed that this rural village was inhabited by artisans and farmers from dark ages until the 14th century, who travelled widely and traded their goods. The people there made widespread use of money and owned beautifully handcrafted ornamental items. The site was likely established for the village due to its strategic position, which allowed its inhabitants to control the communication routes from Como and Varese to Riva San Vitale, Monte Ceneri and the central Alps. Excavations have brought to light a labyrinthine and well-fortified settlement that offered its inhabitants refuge from Barbarian invasions and the skirmishes between Como and Milan.

Why Augmented Reality?

These are stories that Alfio and others excavating the site wanted to share, along with Mendrisiotto Turismo. Initially Alfio and Nadia Lupi, the Director of Mendrisiotto Turismo, envisioned turning Parco Archeologico di Tremona-Castello into a open air, living history museum.

They did not get approval for an open air, living history museum, however, so had to think of alternative ways of helping others to envision life in Parco Archeologico di Tremona-Castello in medieval times. Thanks to 3D Artist and Animator Elia Marcacci volunteering on the excavation of Parco Archeologico di Tremona-Castello, Alfio and Nadia got the idea to have Elia create 3D animations of life in Parco Archeologico di Tremona-Castello in medieval times, which gave them the idea to create an augmented reality experience that would allow visitors to Parco Archeologico di Tremona-Castello to envision the site in medieval times, while still seeing the present day excavation.

The Augmented Reality Experience

In creating this augmented reality experience, Alfio, Nadia and Elia worked with the University of Applied Sciences and Arts of Southern Switzerland (SUPSI) to help shape and technically guide this experience.

They wanted to create an experience that would allow visitors to envision and experience life in the ancient village, while still seeing the excavated foundations of the ancient walls on the site today.

Initially Nadia had imagined this being experienced in AR through a smartphone or a tablet, with ghostly walls of the village and ghostly figures of villagers wandering about around you, going about their daily work and life.

While they do not have that experience yet, they do have an 10 detailed 3D scenes into the past, beautifully illustrated by Elia, including places where you can visualise the ancient village all around you.

The Medieval Village that once stood at Parco Archeologico di Tremona-Castello, as depicted by Elia Marcacci.
The Medieval Village that once stood at Parco Archeologico di Tremona-Castello, as depicted by Elia Marcacci.

Bordering on Italy there was concern that if this experience were made accessible through smartphones and tablets, then visitors would run into issues with their devices accidentally switching to Italian telecommunication networks and resulting in additional charges. Thus the AR storytelling was created to be experienced through AR glasses instead.

To make this a richer experience, sharing audio storytelling about the ancient village and Parco Archeologico di Tremona-Castello, the AR glasses were paired with an audio guide that both shares stories on the 10-minute hike from the visitors centre to Parco Archeologico di Tremona-Castello and within Parco Archeologico di Tremona-Castello.

Due to the AR glasses and the audio guide, visitors can only partake in this experience when the visitor’s centre is open. You can get the address and hours of the visitor’s centre by clicking the button below, as well as details on renting the AR glasses and the audio guide.

Plan for an AR Walk through Parco Archeologico di Tremona-Castello

My Reflections on this Augmented Reality Experience

Part of my goal in exploring different extended reality experiences is to discover what excites me in these experiences and what I find challenging, in order to reflect on what could create even richer and more accessible extended reality experiences.

Here are my thoughts on Parco Archeologico di Tremona-Castello’s AR experience, from when I visited it in 2019.

The Extended Reality Magic

The concept video for Parco Archeologico di Tremona-Castello’s AR experience that Elia, Alfio and Nadia designed is beautiful and would make for an incredible experience for visitors, immersing them in two times / worlds simultaneously through ghostly echos from the past.

Elia’s 3D imagery is also well researched, well thought out, beautifully illustrated, and depicts incredible detail.

A household scene, as depicted by Elia Marcacci, in in the Medieval Village once at Parco Archeologico di Tremona-Castello.
A household scene, as depicted by Elia Marcacci, in in the Medieval Village once at Parco Archeologico di Tremona-Castello.

Based on memory, the audio storytelling was also well done, to transport visitors through time and help them to better imagine life in the ancient village at Parco Archeologico di Tremona-Castello.

Current Challenges (as of 2019 Experience)

As Nadia mentioned, the Parco Archeologico di Tremona-Castello’s AR experience is not yet as she envisioned it in their concept video. In 2019, rather than seeing an active ghostly village around you, you can visit a number of 3D graphic illustrations of life in the village that are generally static, aside from minor movement, like a wisp of smoke. While these 3D illustrations are well done, when you are expecting to walk into an active ghostly village, this is disappointing.

For me, however, the most problematic part of this AR experience is the AR glasses. They are uncomfortable, are difficult to see through, especially if you yourself wear glasses, and are difficult to see the 3D scenes through. As such, visitors don’t get to fully appreciate how well Elia’s illustrations have been crafted.

Erica wearing AR glasses at Tremona Castello Archaeological Park

While the audio storytelling is well designed, the audio storytelling device is problematic, as it has to be set in the visitor’s centre and once you start it, it cannot be paused or replayed without returning to the visitor’s centre to have it reset. This means you can’t take the time to enjoy and immerse yourself in the Parco Archeologico di Tremona-Castello’s AR experience at your own pace, but must stick to the pace of the device. This also means that on a hot, sunny day, you can’t take breaks from the tour under the shade of a tree, but must stay in the sun, until the tour reaches its completion.

The AR glasses and audio guide also limit when visitors can enjoy the experience to the hours of the Visitor’s Centre.

Takeaways From This Extended Reality Experience

Nadia’s initial description of what she envisioned for Parco Archeologico di Tremona-Castello’s AR experience is similar to what I’ve been envisioning for the natural and cultural history augmented reality experiences that I’ve been dreaming of. It is exactly the experience that I was hoping for, and I suspect that with advancements in motion capture suits and world building becoming more accessible through game design engines, like Unreal Engine and Unity, it is becoming easier to make visions like this a reality in the future, making me excited for a future visit to Parco Archeologico di Tremona-Castello to see how the AR experience has evolved. It would seem that Elia has already constructed and made it possible to explore the outdoor spaces of the ancient village that once stood at Parco Archeologico di Tremona-Castello through his 3D animations. Having said that, I think tackling this in pieces of starting with specific scenes, rather than a whole ghostly village all at once, makes this much more manageable logistically and financially, as well as allowing for the opportunity to test drive and learn from the initial scenes before designing more.

The AR glasses were very problematic for me, however, and create an accessibility challenge and I imagine a large expense in establishing this experience. I personally would design a version of this that visitors could enjoy through their smartphones and tablets. With advances in smartphone and tablet technology, the concerns for visitors’ devices switching to an Italian telecommunications network, resulting in unexpected cellular charges, could be resolved in the following ways:

  • Creating a GPS triggered app for the AR experience, that visitors download before leaving the visitor’s centre. Visitors can then turn on airplane mode on their devices, and start the app while using the visitor’s centre wifi. A number of travel audio guide apps work this way, including the popular GuideAlong audio tours.
  • Create a WebAR version of the AR experience, and install a mobile wifi router (like a starlink mobile) at Parco Archeologico di Tremona-Castello for visitors to use while their devices are in airplane mode.

Being able to enjoy this AR experience through a smartphone or tablet would make it accessible to more people, would make Elia’s 3D illustrations easier to view and enjoy, and would allow people to enjoy the experience at their own pace, pausing and replaying it, if they wish. In this way, the AR experience could even be made accessible outside of the visitor’s centre’s hours, if so desired.

It should be noted that one advantage to AR glasses over WebAR is that the augmented realities resolution is usually better with AR glasses, but if the augmented reality is difficult to see through the AR glasses, then it doesn’t really matter how good the resolution is.

Food storage in the Medieval Village once at Parco Archeologico di Tremona-Castello, as depicted by Elia Marcacci.
Food storage in the Medieval Village once at Parco Archeologico di Tremona-Castello, as depicted by Elia Marcacci.

Summary of Takeaways:

  • creating a ghostly ancient village that can be viewed with augmented reality with ghostly figures actively taking part in village life is where the magic in this experience is for me
  • creating a series of ghostly scenes from the village is a more manageable starting point, and allows for the opportunity to test drive and learn from the initial scenes before designing more
  • AR glasses are uncomfortable for some and difficult for some people to see the augmented reality scenes in
  • AR and audio storytelling is more accessible when experienced through a smartphone or a tablet
  • concerns of smartphones or tablets switching to an Italian telecommunications network, could be resolved by having the AR experience accessible through a GPS triggered app or available through WebAR if a mobile wifi router was available for use at Parco Archeologico di Tremona-Castello for visitors to use while their devices are in airplane mode
  • being able to enjoy this AR experience through a smartphones or tablet would allow people to immerse themselves in the experience at their own pace, pausing and replaying it, if they wish

Future Building

In reflecting upon Parco Archeologico di Tremona-Castello’s AR experience, in addition to the desire to have this experience accessible through a smartphone or table, allowing me to pause and replay the audio storytelling, and to see the ancient village in ghostly form with active ghostly villagers, I’d also love to be able to interact with those scenes in some way. Some possible ideas for that:

  • the ability to take pictures with the ghostly village and villagers, including selfies
  • being able to listen in on the conversations of ghostly villagers
  • being able to learn an activity from the ghostly villagers, perhaps as they teach a ghostly child or apprentice
  • being able to partake in a festive event, by dancing or signing along with the ghostly villagers

The educator in me is also envisioning creating the village in VR for use in school classrooms, or possibly VR (or AR) teaser experiences that would get school kids and other visitors prepared for a visit with a bit of a background of the time period that they will be transported to and some questions or things to look out for and think about on their visit to Parco Archeologico di Tremona-Castello.

Exploring Tremona-Castello Archaeological Park with AR glasses.

What’s Your Take on this Augmented Reality Experience?

I’d love to hear your thoughts in comments below on what you think would make this a richer and more accessible AR experience, and if you have technical solutions for making this a more immersive and user friendly experience.

References

Hargreave, E., [StoryToGo]. (2023b, December 15). Tremona Castello Archaeological Park AR Experience [Video]. YouTube. https://www.youtube.com/watch?v=6f8NZaeC_do

La storia. (n.d.). https://aram-ti.ch/il-villaggio/la-storia/

Marcacci Elia. (2014, January 6). Ein Tag in Tremona-Castello. Medienarchiv. https://medienarchiv.zhdk.ch/entries/63853132-1225-498b-ba50-c0b38d3a9eb9

Martinelli, A. (2008). Tremona Castello. Dal V millennio a.C. al XIII secolo d.C. All’Insegna del Giglio.

MendrisiottoTurismo. (2016a, May 25). Il villaggio di Tremona-Castello in 3D [Video]. YouTube. https://www.youtube.com/watch?v=4gzp_A9ADrQ

Mendrisiotto Turismo. (2016b, August 10). Parco archeologico di Tremona-Castello, oggi [Video]. YouTube. https://www.youtube.com/watch?v=ri7HMuhSrVQ

MendrisiottoTurismo. (2016c, August 29). Anno Domini – Rievocazione storica al Parco archeologio di Tremona-Castello [Video]. YouTube. https://www.youtube.com/watch?v=hkCdTiknS-4

MendrisiottoTurismo. (2019a, May 15). Parco Archeologico di Tremona Castello [Video]. YouTube. https://www.youtube.com/watch?v=qHGKHO4bZBI

MendrisiottoTurismo. (2019b, July 12). Hunting for mysteries in time [Video]. YouTube. https://www.youtube.com/watch?v=Ty9LxvWVsOw

Mendrisiotto Turismo. (2023, May 19). La Regione da scoprire – Il Parco archeologico di Tremona 2023 [Video]. YouTube. https://www.youtube.com/watch?v=6BbQJVkru30

Parco Archeologico di Tremona-Castello (n.d.). Ticino Tourism. https://www.parco-archeologico.ch/en/

Società editrice del Corriere del Ticino SA. (n.d.). Tremona – TicinoTopTen. Ticino Top Ten. https://www.ticinotopten.ch/en/experiences/tremona

Exploring XR Development with my Final Master of Educational Technology Directed Studies

by 4 Comments

Join me in exploring XR Development for my final Master of Educational Technology Directed Studies, and finding ways of making XR Development more manageable and accessible for other educators, students and creatives navigating XR and beginning to develop their own projects.

One of my main goals in embarking upon my Masters of Educational Technology was to explore newer-to-me forms of storytelling that could help to bring education to life. Over the past 7 years as I’ve worked my way through my Masters, being introduced to UBC’s Emerging Media Lab and the work they are doing there, having been approached by a board member from the Juno Beach Centre to explore newer ways of sharing stories from World War II that would connect with Canadian school children, travelling to Normandy to visit the museum and see how they are sharing stories there and identifying the gaps in their storytelling, and being shortlisted and interviewed for an interactive and immersive storytelling fellowship in Norway and exploring the stories I might wish to create for that fellowship, the idea of delving deeper into education and storytelling through XR – Extended Reality, teased at my synapses. I took an early course in VR in Education, attended XR symposiums and showcases, began researching and exploring different applications of XR in education and storytelling, collaborated on an open course in Immersive Experiences in Natural and Cultural History Education, developed an idea for an AR storytelling app to share natural and cultural history experiences, and am working my way through other people’s courses in XR, including the University of Michigan’s Extended Reality series of courses. While I am only now beginning to write about all of what I have learned here, the series of posts that follow come from research and a rabbit hole that I began to travel through 6-years-ago, and to date have included four talks at national and international conferences in education, technology, and storytelling, sharing my findings and explorations, and one talk to a Journalism Masters Class at the American University of Cairo. I have shared that last talk below as it shares many of the goals that I set out to accomplish with this directed studies.

Defining Immersive and Interactive Media

Before we get into where my interests and observations are taking me with my final directed studies for my Masters, I thought we’d explore the terminology around immersive media, experiential media, and extended reality, as there is ambiguity around the terminology, so by addressing this at the onset, I hope to at least make it clear as to how I am using the terminology.

An Immersive art exhibit entitled, Interactive Diorama - Rembrandt 1632.
Interactive Diorama—Rembrandt, 1632, The Anatomy Lesson of Dr. Nicolaes Tulp / Aalto University, School of Arts, Design and Architecture, Helsinki. Photographed by Tom Mesic.

Immersive and Interactive Media / Experiential Media

Immersive and Interactive Media are basically the current term for what was termed transmedia, referring to media that is immersive and interactive in nature, allowing those that are engaged with it to experience it in a way that allows them to step into the experience and / or become a part of the media and / or interact with the media, potentially having an impact on how the experience is shaped.

Examples of forms of immersive and interactive media include virtual reality, augmented reality, escape rooms, 360 video, choose your own adventure stories, alternate reality games … and the list goes on. Another name for this form of media is experiential media

Extended Reality (XR)

Project Lead Olivia How, demonstrating Shakespeare xR's touch table of Shakespeare’s First Folio to interested onlookers.

Extended Reality is a form of immersive / experiential media that takes users into a new reality or places virtual objects into a user’s real world, enhancing everyday life with technology. Basically making impossible sensorial experiences, possible. (Hargreave, 2021a)

Included within extended reality are virtual reality (VR), augmented reality (AR), and mixed reality (MR).

Virtual Reality (VR)

Virtual Reality is a computer-generated environment with scenes and objects that appear to be real, making the user feel they are immersed in their surroundings. (Iberdrola, n.d.) While this often is experienced through the use of a headset, a user could also walk into a virtual reality environment created by projectors and screens. This simulated environment and experience could be similar to or completely different from that of the real world in which a user is physically situated.

Examples of virtual reality include 360 video, when experienced through a headset, meeting in Mozilla Hubs, and the Immersive Van Gogh Exhibit.

A student experimenting with the Occulus Quest in the Emerging Media Lab at UBC.
A student experimenting in VR at the Emerging Media Lab at UBC.

Augmented Reality (AR)

Augmented Reality (AR) is an interactive experience where the real, physical world is digitally augmented. The augmentation is usually visual in current applications, but it can also be through other senses, such as auditory or haptic. (Wong, 2021)

Examples of augmented reality include Pokemon Go, the ReBlink exhibition at the AGO, and selfie masking filters in SnapChat.

Exploring Tremona-Castello Archaeological Park with AR glasses.
Exploring Tremona-Castello Archaeological Park with AR glasses.

Mixed Reality (MR)

There is a lot of debate around what constitutes Mixed Reality. I have always thought of it and defined it as allowing for real and virtual elements to interact with one another, and for users to interact with virtual elements in a similar way they would in the real world or at least in a similar way that they might interact with a touch screen. (Hargreave, 2021b)

Students observing the Holobrain at UBC.
Observing the Holobrain at UBC, care of Kerry Blackadar.

Others have defined mix reality (Speicher et al, 2019) as:

  • a synonym for augmented reality
  • a combination of AR and VR
  • a stronger version of AR
  • a type of collaboration
  • the idea of aligning physical virtual environments. 
  • the spectrum from the real environment without any augmentation to the virtual environment, which is completely synthesized computer-generated virtual content. 

This is further confused by some researchers only working within the theoretical and not experimenting with the tools in real time and some corporations attempting to use the term ‘mixed reality’ to market the next level advances in extended reality.

Due to all the debate around defining mixed reality, and based on one research paper, the unlikelihood of a consensus between the various experts in research and industry, I will likely avoid the term mixed reality and instead describe the characteristics of various extended reality applications, and possibly where they lie on the reality–virtuality continuum developed by Paul Milgram and colleagues in 1994.

Reality-Virtuality Continuum

The Reality-Virtuality Continuum as proposed by Paul Milgram, Haruo Takemura, Akira Utsumi, and Fumio Kishino in their 1994 paper, encompasses all possible variations and compositions of real and virtual objects.  They hypothesized that everything between real and the virtual is mixed reality, and that mixed reality makes up both augmented reality and augment virtuality. (Milgram et al, 1994) Basically suggesting that mixed reality is the blending of the physical and the virtual world with varying degrees of augmentation. 

The problem with the RV Continuum in its current form is that it only takes into account visual, not sound, olfactory or haptics senses. (Speicher et al, 2019) It ultimately needs to be redeviced with how technologies are evolving.

Augmented Virtuality

Augmented Virtuality refers to perceiving mostly virtual content, while still seeing some real world content. (Nebeling, 2022)

My Interests and Explorations in Extended Reality

Virtual Reality

I have long been fascinated with extended reality, ever since I was first introduced to the concept of virtual reality in an episode of Murder She Wrote in 1993. It excited me to imagine how extended reality could be used to create immersive story experience and bring worlds to life that are created in one’s imagination. The problem for me though was that whenever I tried it, which I did over the years whenever an opportunity present itself, was that it was too much sensory overload for me, causing mr to feel motion sick (which I now know is cybersickness) and sparked migraines. (Weech et al, 2019)

Jessica Fletcher in a VR headset as she solves a 1993 Murder She Wrote mystery.
Jessica Fletcher in a VR headset as she solves a 1993 Murder She Wrote mystery.

Persistence pays off though, as in 2018 I finally had a VR experience that did not spark cybersickness or a migraine. The experience in question was a virtual plank walk designed to help people overcome their fear of heights. (Basbasse et al, 2023) In exploring the features that made the difference in this experience for me and talking to one of the designers of the experience, I believe the quality of the filming / graphics made the main difference in this experience, as well as perhaps the experience being designed to limit your time in VR. This also got me inquiring from the designer if VR was being used in other ways in the medical world, and if perhaps it was being used with pain management. My interest? By this point I’d been living with chronic pain following two car accidents for 5 years, and was curious if perhaps there were ways this technology could be used to help people better manage their pain. This sparked me to begin explore different medical avenues and uses to VR, as I began to contemplate what might be useful to me and to the community of chronic pain and fatigue sufferers that I now belonged to, many of whom are predominantly home bound.

VR Plank Walk
Epionia Therapeutic VR
Therapeutic VR for Pain Reduction

Augmented Reality

I have always loved and been drawn to educational experiences that incorporate storytelling and encourage the imagination. This was developed in me as a parks naturalist and is part of how I approached science education, and encourage kids to think, imagine, experience and explore.

In approaching work and storytelling, post the car accidents that very much changed my life, how I work, what I can handle and in some ways who I am, I have been increasingly drawn back towards natural and cultural history in my storytelling, as nature soothes my symptoms. When I travel I also love learning and love to envelop myself in the story of a space, but thanks to my post accidents realities I cannot always handle busy museums with noise, lights, and other sensory triggers. This got me imagining, first with my fellowship interview in Norway and later as I explored the Juno Beach Centre in Normandy, if I could create mobile storytelling experiences outside of the museums in places of natural and cultural history, where people could learn and enjoy the stories of the place as they explored outdoors. In both cases, as they were looking to me to imagine newer ways of bringing such stories to life, I began to imagine this with augmented reality as experienced through our mobile devices. This made me smile, as not only did this idea hold the potential for learning and storytelling, but also some of the fun, whimsy and magic that augmented reality brings with it. This is an idea that I further began to sketch out on my travels and in my ETEC 522 Course – Ventures in Educational Technology, and that I am working towards making a reality.

My Goals with this Directed Studies into Extended Reality

I have a tendency to dream big and my dyslexic brain sees connections and expanding and overlapping storyworlds. Usually this means that I do not have the money and often all the knowledge for orchestrating all that I imagine at the get go. My dyslexia has taught me to problem solve though, and life and an imagination and brain that don’t let go of ideas have taught me patience and to play the long game, approaching my ideas in digestible pieces. Thats what I have been doing with this directed studies. I essentially started the work on it after the first year into my Masters, slowly taking advantage of research experiences as they became available, participating in XR symposiums and showcases when they present themselves, taking formal and informal courses in XR, accepting opportunities to speak on what I am working towards with XR and learning along the way, and beginning to flush out my ideas and pitch them to funding bodies.

For this directed studies, my goals are to:

  • Write up the VR case studies that are helping me to identify the strengths and challenges in VR for patient communities.
  • Write up the AR case studies that are helping me to identify the strengths and challenges in AR for natural and history education.
  • Begin to develop and further experiment with VR and AR development.
  • Sharing what I have learned to help make it more manageable and accessible for other educators, students and creatives navigating XR and begin to develop their own projects.

A Beginning

While in some ways I see this directed studies as an end to my researching and a beginning to my developing, there will always be more research to be done in seeing how others are developing and applying the XR, especially as the technology is still developing and despite its decades of history, still very much in its early days for development, application and adoption.

Development-wise, waiting on gear and difficulties with my health and post concussion syndrome, have very much slowed down my intended goals with development and where I’d like to be at this stage, in terms of developing my own XR applications in VR for patients and AR for natural and cultural history. However, as my post accidents self has learned, pushing through when my health issues becomes problematic, only making things worse and setting me back further. Meaning that I have had to be patient with myself, have had to get my health back to a stable place (during a year that posed a lot of challenges with that), and have had to address the backlog of pressing deadlines from my health going sideways before I’ve been able to engage in development. I’ve put in the work to do that, and am now in a good and healthy space to embark on development. This is much later than I would have liked, but that is not something I can change with my newer disability realities post the car accidents. As such, my initial development goals for this directed studies will be smaller for now, and I will continue working on the bigger goals, post the directed studies and will continue to write those up as I do.

Erica wearing AR glasses at Tremona Castello Archaeological Park

This also means that for the Guide to XR Development that I will be developing to make XR Development more manageable and accessible for other educators, students and creatives, this will similarly take shape in stages and over time, some coming sooner with my directed studies and some continuing over time, as I further delve into XR development and experiments of my own.

Thank you for joining me on this journey into XR Development, and being patient with me as I take the time to do this in a way that is healthy for me with my health and disability realities. Hopefully my journey as the tortoise on this road will inspire others to explore and embark on such a journey of their own, no matter at what speed and working with whatever challenges your life and situation might pose along the way.

References

El Basbasse, Y., Packheiser, J., Peterburs, J., Maymon, C., Güntürkün, O., Grimshaw, G., & Ocklenburg, S. (2023). Walk the plank! Using mobile electroencephalography to investigate emotional lateralization of immersive fear in virtual reality. Royal Society open science, 10(5), 221239. https://doi.org/10.1098/rsos.221239

Hargreave, E. (2021a, November 15). An introduction to immersive experiences in natural & cultural history education – StoryToGo Classroom. StoryToGo Classroom. https://storytogo.ca/classroom/course/immersive-experiences-in-natural-and-cultural-history/lessons/intro/

Hargreave, E. (2021b, November 15). Mixed Reality (MR) in Natural & Cultural History Education – StoryToGo Classroom. StoryToGo Classroom. https://storytogo.ca/classroom/course/immersive-experiences-in-natural-and-cultural-history/lessons/mr-in-natural-cultural-history-education/

Hargreave, E., [Stories, EdTech & Digital Media with Ahimsa Media]. (2020, November 29). Story Steppers – an AR storytelling app to the natural & cultural history of a place [Video]. YouTube. https://www.youtube.com/watch?v=2xeWjBWYcAI

Hargreave, E., [StoryToGo]. (2023, December 10). Exploring Immersive Media to Develop Creative & Engaging Stories with Master of Journalism Students [Video]. YouTube. https://www.youtube.com/watch?v=Ff1cMMiH5cc

Iberdrola. (n.d.). Virtual Reality: another world within sight. Retrieved December 7, 2023, from https://www.iberdrola.com/innovation/virtual-reality

P. Milgram, H. Takemura, A. Utsumi, F. Kishino: Augmented Reality: A class of displays on the reality-virtuality continuum. In Proceedings of Telemanipulator and Telepresence Technologies, 1994.

Nebeling, M. (2022, June 2). Intro to AR/VR/MR/XR: Technologies, applications & issues. Coursera. https://www.coursera.org/learn/intro-augmented-virtual-mixed-extended-reality-technologies-applications-issues

M. Speicher, B.D. Hall, M. Nebeling: What is Mixed Reality? In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI 2019, Glasgow, Scotland, UK, May 4-9, 2019.

Weech, S., Kenny, S., & Barnett-Cowan, M. (2019). Presence and Cybersickness in Virtual Reality Are Negatively Related: A Review. Frontiers in psychology, 10, 158. https://doi.org/10.3389/fpsyg.2019.00158

Wong, Y. (2021, November 15). Augmented Reality (AR) in Natural & Cultural History Education – StoryToGo Classroom. StoryToGo Classroom. https://storytogo.ca/classroom/course/immersive-experiences-in-natural-and-cultural-history/lessons/ar/