StoryToGo

Exploring storytelling in its many forms in this digital age

Extended Reality

Conceptualising an Extended Reality Open Education Guide

by 6 Comments

As we step into 2024 and I wrap up my ETEC 580 Directed Studies and Master of Educational Technology, I want to take a moment to conceptualise the Extended Reality Open Education Guide that I have begun to build; to give it shape, intention and create a road map for it, as well to seek feedback from you, our readers, as I continue to further my journey, experiments, and goals in extended reality (XR) beyond my Masters. As I previously wrote in Exploring XR Development with my Final Master of Educational Technology Directed Studies, the work I have done through the MET Program at UBC is a beginning to this work in XR development.

Here’s where I am at present in mapping out where I am going with this work:

XR Open Education Guide for StoryToGo

As I continue to develop an XR Open Education Guide on StoryToGo, much as I did with the Web Monetization Guide, there are 4 main areas that I will be focusing on, developing, and building upon over time:

  • XR Case Studies
  • XR Development Experiments and Process Blogging
  • XR Development Series of Mini-Courses
  • Addressing Accessibility in Extended Reality

XR Case Studies

The goal of the Extended Reality Case Studies is to explore different extended reality experiences and applications to discover what resonates with myself and my team and to identify the challenges. By writing these up as case studies, the hope is to create a resource for our team in developing extended reality experiences, and to share that resource more broadly with other independent creatives, educators and students wanting to explore extended reality. Secondarily, these XR Case Studies will also be used as resources and examples within the XR Development Series of Mini-Courses.

At present, these XR Case Studies are focused around augmented reality (AR) natural and cultural history experiences and virtual reality (VR) health and wellness experiences, as these are the areas that my team and I are currently developing extended reality projects. Over times, we will broaden these case studies to other areas of interest.

Extended Reality Case Studies

As my team’s and my current priority is on AR natural and cultural history experiences, the current case studies include:

I am also writing up a case study on the TaleBlazer AR Game at the Royal Botanical Garden’s Rose Garden at present.

Other XR Case Studies on the current ‘to write’ list, include:

  • Notre-Dame de Paris, The Augmented Exhibition
  • Plank Walk VR
  • Pain Distraction VR
  • Concussion Therapy VR

As the XR Case Studies grow in number on StoryToGo, we will create an Experiential Reality Library, like the Web Monetized Content Library that we’ve begun to create.

XR Development Experiments and Process Blogging

As an indie creative who likes experimenting with emerging ways of crafting stories, I find it helpful to experiment with the technology for building those stories. That way even if we contract a larger team on our builds, I know what I am talking about and know what is possible. Often as I dive into such experiments, I also learn that building stories in these newer ways with emerging technologies is not so daunting, cost prohibitive, and inaccessible as people think. Thus, as I create and experiment with extended reality development, using different extended reality development tools and platforms, I intend to create process blogs walking people through the ‘how to’ of my experiments creating with different technologies.

The purpose?

  • to create a running log / record of the process to pull from for the XR Development Series of Mini Courses;
  • to engage others around this process, and crowdsource ideas and solutions; and
  • inspire others to experiment with their own XR builds, and provide them with a resource to helping them do so.
Extended Reality Development Experiments and Process Blogging

Currently my goals for my own projects involve Unreal Engine and motion capture technology. This, however, is not the most accessible entry point for educators and students wanting to begin their own experiments with XR, both in terms of the learning curve and in terms of the expense of the technology needed for. As such, I decided for the purposes of creating a more accessible entry to extended reality for teachers and students, that I want to begin with some more accessible extended reality builds / development ideas, and then build towards the more involved pieces using UnReal Engine and Unity, as part of my long term, on-going work.

What this means, is that my initial AR build will be with TaleBlazer to create some early development options within the open Extended Reality Course that are designed for teachers and students, and then later build Units on developing with UnReal Engine and Unity, as well as other more accessible extended reality development solutions, like Stornaway.io, CoSpaces, Halo AR, polycam, MyWebAR, and 8th Wall. If you have other suggestions of XR development tools and platforms that my team and I should be experimenting with, please let me know about them in the comments on this post.

In terms of the extended reality projects that my team and I have been developing, they include:

  • WWII stories from my friend Manami Calvo (Saito)’s family from their experience as Canadians of Japanese ancestry, living on the West Coast in WWII
    • our plan for this is to create some initial experimental ‘art installations’ using Web AR, that are accessible via smartphones and tablets
    • we plan on creating these utilizing animated ‘ghostly figures’ created with Unreal Engine and a motion capture suit, along with historic photos and video, as well as recorded interviews
    • we will then use those art installations to go after proper funding to further development

  • health / wellness VR Experiences
    • our plan is to start with a kids yoga class in VR that can be used in hospitals, as well as at home
    • create in Unreal Engine with a motion capture suit
    • the yoga will be lead by Lori Yearwood (wearing the motion capture suit) and will involve her transforming to different plants and animals with different poses
    • the children will also be able to select the background world (from a variety of calming choices) for their yoga experience

XR Development Series of Mini-Courses

To make things as easy as possible for other indie creatives, educators and students to begin to explore and develop their own extended reality projects, I intend to take what I have learned through my extended reality research, case studies, experiments and process blogging, and build a series of Extended Reality Mini-Courses for people to use as a guide in creating their own extended reality projects. The goal here being to remove the fear of extended reality being a far reaching, inaccessible goal by building their understanding of extended reality, getting them experimenting with it and envisioning the possibilities, and sharing with them a variety of pathways to creating extended reality, including step-by-step guides.

Extended Reality Development Courses

In developing a series of Extended Reality Mini-Courses, I am exploring other Extended Reality Learning Resources to see what is currently available to those wishing to explore extended reality further, so as best to identify a different, unique approach to the subject and include any beneficial resources that are currently missing.

The Extended Reality Learning Resources that I am currently delving into, include:

Pre-Existing Extended Reality Courses

In approaching the extended reality course development for the StoryToGo Classroom site, I have come to the conclusion that XR development is more manageably approached as a series of mini-courses. Currently, I am thinking in terms of building the following three courses:

  • First Steps in Extended Reality Development – An Introduction
  • Extended Reality Development
  • Extended Reality User Testing and Refining

The talk I gave for the class at the American University in Cairo follows the layout for the first mini-course: 

My current breakdown for ‘First Steps in Extended Reality Development – An Introduction’ will include:

  • What Extended Reality Is
    • types of Extended Reality
    • applications of Extended Reality
    • delivery of Extended Reality and Devices

  • Extended Reality Case Studies
    • an informal introduction to user testing
    • creating mini case studies in user testing extended reality experiences, and recording stars and wishes from the experiences
  • Developing an Extended Reality Idea of Your Own
    • Why XR?
      • what type of XR?
      • is this the strongest choice?
      • how do you wish to deliver? to which devices? why?
    • Do you need to create all at once or can you create it in accessible steps?
    • Steps in creating
    • Creating a roadmap

Then the Extended Reality Development Course will be a continual work in progress, adding different Units, covering how to build XR with different extended reality development solutions, starting with TaleBlazer.

Extended Reality Mini Course to Develop for StoryToGo

If you have suggestions for the structure of or things I should be including in the XR Development Series of Mini-Courses, please share your thoughts in the comments.

Addressing Accessibility in Extended Reality

While not a standalone piece on my brainstorming document, for those of you who have begun to read our XR Case Studies, you may have recognized that accessibility in extended reality is central to my team’s and my explorations and development goals. This is very much driven by various disabilities and health and wellness issues that my team and I have experienced, but is also an aspect of extended reality design and user testing that has not been discussed and addressed as well as it should yet.

For now, my team and I intended to address Accessibility in Extended Reality throughout our case studies, process blogging, and course materials. I also had an invitation to come on as an advisor on accessibility to UBC’s Emerging Media Lab, after I brought accessibility and my perspective on with a few of the students at their Autumn Showcase. This is an invitation I intend to follow up on this year, with the hope that between that and my own team’s process thinking through accessibility in extended reality, as we share extended reality case studies, work on our own extended reality development and process blog on it, and create our Extended Reality Development Series of Mini-Courses, that we also begin to build a standalone piece or pieces that address accessibility in extended reality. Not sure what that will be yet, but if you have suggestions for this, I’d love to hear about them in the comments.

Your Thoughts?

I’d love to hear your thoughts in comments below on the direction I am taking with this Extended Reality Open Education Guide. If you have suggestions for future XR Case Studies, on XR Development Tools and Platforms to try, the structure of or things I should be including in the XR Development Series of Mini-Courses, or ways to further address Accessibility in Extended Reality, please share them in the comments. Thank you!

Also, if anyone wishes to become involved in any of the above endeavours with myself and my team, please reach out and we can chat further.

Thank Yous

Thank You!

The work that I have begun here has been inspired by and is evolving, thanks to:

  • Dr David Vogt for encouraging my dreaming, entrepreneurial thinking, and edtech product design and development;
  • Dr David Porter in exploring open education and searching for sustainable funding pathways for;
  • Saeed Dyanatkar for seeing my potential in dreaming up different pathways for storytelling and education, always making time for me when I had questions, and making me feel welcome and valued whenever I wandered into the Emerging Media Lab;
  • Dr Kyle Stooshnov and Juliana Loh for creating a framework for my first steps into thinking about XR development and experimenting with 360 video;
  • Dr Jennifer Jenson and Dr Suzanne de Castell for fostering my game design thinking;
  • Dr Heidi Janz, Dr Michelle Stack and Dr Paul Hamilton for helping me to process my newer disabilities, frame disabilities and myself as disabled in my storytelling, and begin to understand how to advocate for disability and accessibility in education (and feel empowered to do so, understanding it as a strength, as oppose to something to hide);
  • Yvonne Dawydiak for demonstrating the alternative pathways to navigating UBC, approaching teaching creatively in a hands-on way, and an empathy driven approach and practice;
  • Lori Yearwood for her vision in beginning to create case studies here on StoryToGo to act as resources for our students, workshop participants, clients and partners, and the broader independent creative and education community – for encouraging me to begin to share my disabilities in our storytelling and work, and sharing her own wellness challenges as we began to build our health, wellness and accessibility workshops, resources, and storytelling – and for always being a willing and enthusiastic supporter and co-creator on the ideas that wander through my mind; and
  • Manami Calvo for seeing the potential for sharing her family’s WWII stories in AR, joining me in crafting experiments around, and in trusting me with her family’s stories.

Also an enormous thank you to the Interledger Foundation for helping fund my research and work on Sustainable Funding Solutions for Creatives and Educators, and to CNIE – RCIÉ and BCIT who have helped fund my research and coursework throughout my Masters.

Thank you all, for all your help, support, direction and inspiration!

Seeing the Invisible Augmented Reality Art Exhibit – an Extended Reality Case Study

by 4 Comments

In this second case study in our series exploring extended reality (XR), we examine the Seeing the Invisible Augmented Reality (AR) Art Exhibit, as exhibited by the Royal Botanical Gardens. This AR art exhibition featuring the art of 13 artists from around the world was initiated by the Jerusalem Botanical Gardens and displayed concurrently at 12 Botanical Gardens worldwide.

Project:

Seeing the Invisible Augmented Reality Art Exhibit

Initiated By:

Jerusalem Botanical Gardens in partnership with Outset Contemporary Art Fund, with the support of the Jerusalem Foundation

Curated By:

Hadas Maor and Tal Michael Haring

Host At:

Royal Botanical Gardens (RBG)

Type of Experience:

Augmented Reality (AR) Outdoor Art Exhibition

Device for Experiencing:

Smartphone or Tablet, preferably with headphones

Link to Project:

https://seeingtheinvisible.art/

Date Partook in the Experience:

September 18, 2022

Country of Origin:

Israel

Country Where I Experienced:

Canada

Reflections On My Experience

Interview Between Tucson Botanical Gardens Executive Director Michelle Conklin & Seeing The Invisible Curators Hadas Maor & Tal Michael Haring

The Project:

In 2021, as the world continued to face the pandemic and many art galleries were closed to the public, an idea was fostered out of the Jerusalem Botanical Gardens to host an outdoor art exhibit, but not in the usual sort of way. Rather an invisible art exhibit, at least to the naked eye, and one that was concurrently displayed in 12 gardens worldwide with the same pieces of original art. “How?”, you ask. With augment reality (AR), creating the first such multi-location, simultaneous art exhibition of its kind with a number of artists, many of whom were new to AR, creating their first pieces in augmented reality.

Some of the goals of this exhibition were to:

  • give visitors something new to enjoy in gardens at times of year when plants may not be blooming
  • complement the natural setting with the AR art, encouraging visitors to experience it in a new way
  • give people a way to enjoy art in a shared, yet outdoor setting
  • encourage visitors to the gardens to engage with the gardens and with the art
  • collaborate with other gardens worldwide
  • create an exhibition without disturbing the gardens themselves, and keeping the carbon footprint to a minimum

The art exhibition was also designed to address shared themes of “nature, environment, and sustainability, exploring the boundaries and connections between art, technology, and nature. Both bleak and hopeful, each artwork offering a unique perspective on unresolved issues, creating thought-provoking, experiential, and contemplative spaces for viewers to immerse in.”

The artwork and artists crafting this narrative included:

Despite the art exhibition taking place concurrently in a number of different gardens, the experience is different in each garden, as the works are augmenting the unique surroundings and context of each garden.

Why Augmented Reality?

Building and purchasing sculptures and other art installations is a pretty major undertaking for a botanical gardens, and a travelling exhibition can generally only visit one garden at a time. While not without a cost, by making this art exhibition simultaneously available in a number of botanical gardens, this makes the cost of such an art exhibition less prohibitive. The fact that this art exhibition is accessible through augmented reality via digital devices, makes it possible to ‘install the pieces’ temporarily in multiple gardens at the same time.

The initial launch of this AR art exhibition during the pandemic was well timed to offer an alternative outdoor way to enjoy art, outside the enclosed space of an art gallery, and further encouraged visitors to think about botanical gardens as arts spaces, as well as nature spaces.

The curators of Seeing the Invisible talk about the importance of this AR art exhibition being phygital, a blending of a digital experience with a physical one. Thought was put into where each artwork was placed in each garden to enhance both the artwork and the garden, as well as take visitor safety and the protection of the planted exhibitions into consideration. The AR artwork, through the Seeing the Invisible app, was geotagged to specific locations in the gardens they were exhibited in, and visitors could only experience the artwork in that location, and only in those gardens. Thanks to the phygital, the experience was unique in each garden.

The Augmented Reality Experience

The Seeing the Invisible artwork is experienced at the gardens hosting this AR art exhibition through visitors’ smartphones and tablets via a GPS triggered app.

Viewing AR art, Dawn Chorus, through a smartphone.

To see and experience these dynamic and engaging pieces of artwork, people visiting the gardens hosting this AR art exhibition need to:

  • Before visiting a Gardens hosting the AR art exhibition:
    • Download the Seeing the Invisible App to the smartphone or tablet with cellular capabilities that they will be using at the AR art exhibition.
    • Allow the app access to device’s camera and microphone.
    • Fully charge the device before visiting the gardens.
    • Take earbuds or headphones compatible with the device to the gardens with you.
  • At a Gardens hosting the AR art exhibition:
    • If you don’t have a smartphone or tablet, you can borrow one on site.
    • Follow the map in the app to the different pieces of AR art within the gardens.
    • Follow the Seeing the Invisible App’s instructions at each piece of AR art, scanning the ground where the piece of art has been virtually installed until the art virtually appears.

The Royal Botanical Gardens shares further tips on enjoying Seeing the Invisible Art Exhibition in their gardens, here.

Once the art virtually appears before you, visitors are encouraged to engage with it and fully immerse themselves in it, by listening through their headphones to the experience, and walking around the artwork and even into it. Visitors can also take pictures with the virtual art and read more about the art through the app.

Each piece of art, lends to different forms of engagement. For instance with:

  • Gilded Cage: You can walk into the labyrinth of cells.
  • Dawn Chorus: Birds swoop around you as you are drawn in by the music from the piano.
  • Biome Gateway: You can walk into the temple cave, and discover a portal to walk through and into a parallel landscape.
  • Forget Me Not: By moving toward the flower arrangement, you trigger a gun to propel a bullet through the flower arrangement causing it to explode outwards. Once the explosion has occurred, if you walk around the vase, you will hear three different scholars discussing the flower arrangement.
  • Pneuma: By walking into the sphere, you see the gardens around you through the distortion of being inside a bubble, while the sound of breathing acts to change your own breathing pattern while experiencing.

Morphecore Prototype AR even inspired a dancer to choreograph their own piece to dance along with. Other host Gardens similarly witnessed many of their visitors dancing along with the Morphecore Prototype.

My Reflections on this Augmented Reality Experience

One of my goals in exploring different extended reality experiences is to discover what excites me in these experiences and what I find challenging, in order to reflect on what could create even richer and more accessible extended reality experiences.

Here are my thoughts on the Seeing the Invisible AR Art Exhibition, from when I visited it in September of 2022 at the Royal Botanical Gardens.

Signs welcoming visitors to the Seeing the Invisible AR Art Exhibition at the Royal Botanical Gardens.

The Extended Reality Magic

This is a clever idea to get visitors exploring the gardens through a new lens, allowing the augmented reality art to spark their imagination in a new way, to perceive the gardens differently, and to provoke thought pertaining to nature, technology and art.

Some art, like Dawn Chorus, brought pure joy of the fantastically, magical whimsy that augmented reality brings with it, further adding beauty to the surrounding gardens.

AR birds flying around an augmented reality piano in a real world garden.
AR birds flying around an augmented reality piano, care of Dawn Chorus, in a real world garden.

Other art, like Pneuma, made you feel as though you’d stepped out of reality and were peering back in at it.

AR Art looking back at the gardens through the distortion of the augmented reality bubble created by Mel O’Callaghan's Pneuma.
Looking back at the gardens through the distortion of the augmented reality bubble created by Mel O’Callaghan’s Pneuma.

While other art, like Gilded Cage, got you questioning our problematically omnipotent and controlling relationship with nature from the perspective on an artist who was imprisoned.

AR art, Ai Weiwei's golden Gilded Cage, casting augmented reality shadows, by a park bench in a real world garden.
AR art of Ai Weiwei’s golden Gilded Cage, casting augmented reality shadows by a park bench in a real world garden.

Best of all, the gardens discovered that people of all ages wanted to engage with the art, even dancing along to Daito Manabe’s Morphecore Prototype AR.

A real world dancer at Jerusalem Botanical Gardens dancing alongside Daito Manabe's Morphecore Prototype AR.
A real world dancer at Jerusalem Botanical Gardens dancing alongside Daito Manabe’s Morphecore Prototype AR.

Logistically the GPS triggered app is a good idea to create a curated experience within each garden, and to host this simultaneously between different gardens.

Adding in the ability for visitors to be able to photograph the AR art through the app, added in the ability for visitors to have fun engaging with the art. This also added to crowdsourced storytelling around the exhibit, which I can only imagine was beneficial to marketing the exhibition.

Visitors at a number of gardens playing with Pneuma selfies.
Visitors at a number of gardens playing with Pneuma selfies.

I’d love to experience future such AR art exhibitions at the Royal Botanical Garden, as well as to experience the Seeing the Invisible Art Exhibition in different gardens to see how the experience differs between the gardens with the same art.

Current Challenges (as of 2022 Experience)

Excited to experience the Seeing the Invisible Art Exhibition at the Royal Botanical Gardens in Burlington, Ontario, my niece and I followed the instructions and downloaded the Seeing the Invisible App to our smartphones before heading to Hendrie Park to enjoy the exhibit together.

Seeing the Invisible at the Royal Botanical Garden's Hendrie Park.

The only problem, when we got there, the Seeing the Invisible App didn’t work on either of our smartphones, despite both having compatible technology. Not sure if this is the reason why, but one woman told us the issue was due to the strength of the wifi signal from your data provider.

No problem, we queued to borrow one of tablets that the gardens had available for visitors that needed it.

Borrowing a loaner tablet at the Royal Botanical Gardens.

While the borrowed tablet worked, this now meant that we were now sharing a device, our headphones didn’t work in the borrowed device, and the borrowed tablet had a screen protector over it, that made the AR more difficult to see, especially in sunny areas. Added to this, almost every piece of AR art was placed in a sunny area, and it was hot, so not only were we battling with the light to see the AR art through the screen, but needed shade for respite for the sun.

I assumed that this user experience oversight was down to this being the Royal Botanical Garden’s first AR Exhibition and them not being familiar enough with the technology to understand how to create a positive user experience, but it turns out it was actually the curators that mapped out where each piece would be placed. Maybe the oversight of the sun was made, as due to the pandemic, the curators did not visit Hendrie Park before deciding where to place the AR art, but either which way, this emphasizes the importance of taking the user experience into consideration when designing AR exhibitions. Shade is important, both for the visitors’ physical comfort and for the practicality of visitors actually being able to properly see and experience the AR art work through their screens. With the size of the Royal Botanical Gardens there is more than enough options to place these 13 pieces of artwork in a way that keeps visitors out of the full sun and allows them to better see and experience the AR art on their screens.

Girl holding a tablet, demonstrating the difficulty seeing the AR art in the sun.

While certainly the audio experience would have been much better had we been able to use our headphones, as people do experience exhibits like this on a shared device, I think it would be beneficial for the audio portion of the experience to be added at a higher volume to better allow for visitors sharing a device to be able to hear the experience.

I appreciated that the artist shared a write up on their art within the app, but to spend time reading this, it takes you away from viewing and experiencing the AR art. It would be great, if visitors have the additional option to play an audio recording of the artist sharing their thoughts with visitors on the piece of art.

Finally, with the loaned tablets, it would be beneficial if the individual gardens made it possible for visitors to email themselves the copies of any photos they shot with the AR art.

Takeaways From This Extended Reality Experience

When easily visible and heard, the Seeing the Invisible Art Exhibition creates a glimpse into a hidden world that encourages thought and exploration through a new lens and perspective. Thats exciting, and demonstrated what excites me about sharing stories with augmented reality.

However, if the app isn’t working on people’s devices or the AR is difficult to see on people’s screens, then people become frustrated, rather than excited by the experience. This is why user design and testing is so very important. This should be thought of both in terms of the technology and visitors’ interactions with the physical environment in which they are experiencing the AR art. In the case of the RBG, the gardens and not trampling plants seems to have been taken into considerations, but not visitors’ physical comfort from the sun or thought of how that sun would impact user’s experience. This art exhibition was almost in its entirety in direct sun with no to very little shade.

A girl in another full sun AR art exhibit spot at the Royal Botanical Gardens.

User testing should be inclusive of the loaner tablets for visitor use, made available onsite. This means, if adding protective screen covers to the tablets, making sure those protective screen covers do not hinder visitors’ ability to enjoy the AR art.

It would also be advantageous to allow people using the borrowed tablets to be able to email themselves any pictures they took with the AR art, or if that is not possible, to be upfront with visitors about that from the start, so as not to leave them disappointed at the end of the experience.

Summary of Takeaways:

  • The technology exists to create a GPS triggered augmented reality app that works on people’s devices without overloading a device’s data.
  • The Seeing the Invisible AR App takes up 2.1 GB of space on my smartphone.
  • The Seeing the Invisible AR App was developed by Khora ApS, a virtual reality and augmented reality production studio in Copenhagen.
  • When the AR app is working, the augmented reality art exhibition creates wonder for visitors, gets visitors exploring and interacting in a new way, and has visitors engaging with the AR art.
  • By making the AR art exhibition viewable through smartphones and tablets (with loaner tablets available for use), it is accessible to all ages and most abilities.
  • User testing is important on the site of the experience, thinking about user comfort at different times of year.
  • Avoid creating GPS triggered AR screen experiences in direct sun.
  • Make sure protective screen covers on loaner tablet do not limit the ability to see the augmented reality.
  • Think about user safety and comfort when placing GPS triggered AR experiences.
  • Encourage visitors to bring headphones, while ensuring sound is available at a volume that those without headphones or sharing a device can hear.
  • People enjoy interacting with AR art.
  • Make sure people can email themselves any pictures they took with the AR art on loaned tablets.

Future Building

In reflecting upon the Seeing the Invisible AR Art Exhibition, in addition to placing the augmented reality art in spaces that people can enjoy the art in the shade and to be better able to hear the accompanying sounds when experiencing through a shared device, I’d also love to further encourage the engagement that people enjoyed with the art. Some possible ideas for that:

  • I’ve recently seen how the Relive App makes it easy to create videos from a walk by recording the map and editing in the photos and videos you take at different stops. Should this be possible within this AR app, it could create fun keepsakes from the AR experience, that also become valuable crowdsourced storytelling for the exhibit. An example from the travels of a fellow member of Vancouver’s tech community:
  • It would be great to be able to play the artist’s description of their work as an audio reading, complete with some prompts to encourage people to engage with their art in different ways.
  • I loved how Ori Gersht’s Forget Me Not was responsive to visitor’s movements, it would be exciting to have more pieces that respond to the location of the holder of a device.
Ori Gersht's Forget Me Not AR art flower arrangement.
Exploding vase of flowers, as a part of Ori Gersht's Forget Me Not responsive AR art.

The educator in me has also begun to design, in my head, a scavenger hunt that encourages visitors to collect experiences and different perspectives with each piece of art by encouraging visitors to take on some sort of challenge or unravel some sort of mystery with each piece of art. Being able to create a mapped storytelling account of such a scavenger hunt, in a format like the Relive App creates, would be a great way to further share the discoveries from such a scavenger hunt.

The storyteller in me would also love to learn more about the making of the art, like with this talk on Ori Gersht’s and Timur Si-Qin’s AR art pieces.

What’s Your Take on this Augmented Reality Experience?

I’d love to hear your thoughts in comments below on what you think would make this a richer and more accessible AR experience, and if you have technical solutions for making this a more immersive and user friendly experience.

One visitor using an umbrella to improve his AR art experience at the Seeing the Invisible Art Exhibition at the RBG, with all the sun.
One visitor using a black umbrella to improve his AR art experience in the sun.

References

Conklin, M., [Tucson Botanical Gardens], Maor, H., & Haring, T. (2022, December 1). A Talk with the Curators of Seeing The Invisible [Video]. YouTube. https://www.youtube.com/watch?v=1BXqc0g8D_c

Horwitz, L. (2023, June 12). phygital. Customer Experience. https://www.techtarget.com/searchcustomerexperience/definition/phygital

Jerusalem Botanical Garden. (2021, November 14). אחת העבודות בתערוכת האמנות, תפתיע אתכם במיוחד. . .. Jerusalem Botanical Garden’s Facebook Page. https://www.facebook.com/Jerusalem.Botanical.Gardens/videos/372403381331281

Rendell, H., Gertler, C., Katri, M., Maor, H., & Haring, T. (2021). SEEING THE INVISIBLE. SEEING THE INVISIBLE. https://seeingtheinvisible.art/

Royal Botanic Garden Edinburgh, Maor, H., Gersht, O., & Si-Qin, T. (2022, May 24). Seeing the Invisible  In Conversation with the Artists [Video]. YouTube. https://www.youtube.com/watch?v=3_XD4nmV-58

Royal Botanical Gardens. (2023, March 8). Seeing the Invisible – Royal Botanical Gardens. https://www.rbg.ca/things-to-do/art-in-the-gardens/seeing-the-invisible

Shamir, R., [America-Israel Friendship League], Gertler, C., Rendell, H., Maor, H., Haring, T., Rominiecki, J., & Firestone, W. (2021, November 8). Seeing the Invisible: Augmented Reality art [Video]. YouTube. https://www.youtube.com/watch?v=TIuxE1oZKtY

Parco Archeologico di Tremona-Castello AR Experience – an Extended Reality Case Study

by 8 Comments

Welcome to our new series exploring extended reality case studies, in a quest to develop richer and more accessible Extended Reality (XR) experiences. In this first case study, we delve into Parco Archeologico di Tremona-Castello’s (Tremona Castello Archaeological Park) Augmented Reality (AR) Experience and speak with Nadia Lupi, one of the visionaries behind the project.

Organization: 

Parco Archeologico di Tremona-Castello (Tremona Castello Archaeological Park)

Project:

Parco Archeologico di Tremona-Castello AR Experience

Type of Experience:

Augmented Reality Walking Tour

Device for Experiencing:

Augmented Reality Glasses paired with a guided audio tour

Link to Project:

https://www.parco-archeologico.ch/en/visita/la-realta-virtuale.html

Interview / Partook in the Experience Date:

April 30, 2019

Interview With:

Nadia Lupi, Director of Mendrisiotto Turismo

Country:

Switzerland

Delving Into the Tremona Castello Archaeological Park AR Experience & an Interview with Nadia Lupi

The Project : Parco Archeologico di Tremona-Castello AR Experience

Parco Archeologico di Tremona-Castello

Sitting on top of the hillside over looking the village of Tremona in the Mendrisiotto District of the Canton of Ticino, Switzerland, local archaeologist Alfio Martinelli, discovered the archaeological remains of an ancient medieval village sharing a history of the area for 6,000 years from prehistory to the 14th century. As Alfio and other archaeologists and volunteers began to uncover this village, many questions arose as they began the puzzle together the stories from the past. Among the curiosities of Parco Archeologico di Tremona-Castello are why they chose to build there, with no apparent water source on site.

Who lived here? How did they spend their daily lives? Why did they suddenly abandon this village in the 14th century? These are all questions that Alfio, fellow archaeologists, and volunteers excavating Parco Archeologico di Tremona-Castello have been asking, as they piece together pictures and stories of daily life in this ancient village when it was thriving.

It is believed that this rural village was inhabited by artisans and farmers from dark ages until the 14th century, who travelled widely and traded their goods. The people there made widespread use of money and owned beautifully handcrafted ornamental items. The site was likely established for the village due to its strategic position, which allowed its inhabitants to control the communication routes from Como and Varese to Riva San Vitale, Monte Ceneri and the central Alps. Excavations have brought to light a labyrinthine and well-fortified settlement that offered its inhabitants refuge from Barbarian invasions and the skirmishes between Como and Milan.

Why Augmented Reality?

These are stories that Alfio and others excavating the site wanted to share, along with Mendrisiotto Turismo. Initially Alfio and Nadia Lupi, the Director of Mendrisiotto Turismo, envisioned turning Parco Archeologico di Tremona-Castello into a open air, living history museum.

They did not get approval for an open air, living history museum, however, so had to think of alternative ways of helping others to envision life in Parco Archeologico di Tremona-Castello in medieval times. Thanks to 3D Artist and Animator Elia Marcacci volunteering on the excavation of Parco Archeologico di Tremona-Castello, Alfio and Nadia got the idea to have Elia create 3D animations of life in Parco Archeologico di Tremona-Castello in medieval times, which gave them the idea to create an augmented reality experience that would allow visitors to Parco Archeologico di Tremona-Castello to envision the site in medieval times, while still seeing the present day excavation.

The Augmented Reality Experience

In creating this augmented reality experience, Alfio, Nadia and Elia worked with the University of Applied Sciences and Arts of Southern Switzerland (SUPSI) to help shape and technically guide this experience.

They wanted to create an experience that would allow visitors to envision and experience life in the ancient village, while still seeing the excavated foundations of the ancient walls on the site today.

Initially Nadia had imagined this being experienced in AR through a smartphone or a tablet, with ghostly walls of the village and ghostly figures of villagers wandering about around you, going about their daily work and life.

While they do not have that experience yet, they do have an 10 detailed 3D scenes into the past, beautifully illustrated by Elia, including places where you can visualise the ancient village all around you.

The Medieval Village that once stood at Parco Archeologico di Tremona-Castello, as depicted by Elia Marcacci.
The Medieval Village that once stood at Parco Archeologico di Tremona-Castello, as depicted by Elia Marcacci.

Bordering on Italy there was concern that if this experience were made accessible through smartphones and tablets, then visitors would run into issues with their devices accidentally switching to Italian telecommunication networks and resulting in additional charges. Thus the AR storytelling was created to be experienced through AR glasses instead.

To make this a richer experience, sharing audio storytelling about the ancient village and Parco Archeologico di Tremona-Castello, the AR glasses were paired with an audio guide that both shares stories on the 10-minute hike from the visitors centre to Parco Archeologico di Tremona-Castello and within Parco Archeologico di Tremona-Castello.

Due to the AR glasses and the audio guide, visitors can only partake in this experience when the visitor’s centre is open. You can get the address and hours of the visitor’s centre by clicking the button below, as well as details on renting the AR glasses and the audio guide.

Plan for an AR Walk through Parco Archeologico di Tremona-Castello

My Reflections on this Augmented Reality Experience

Part of my goal in exploring different extended reality experiences is to discover what excites me in these experiences and what I find challenging, in order to reflect on what could create even richer and more accessible extended reality experiences.

Here are my thoughts on Parco Archeologico di Tremona-Castello’s AR experience, from when I visited it in 2019.

The Extended Reality Magic

The concept video for Parco Archeologico di Tremona-Castello’s AR experience that Elia, Alfio and Nadia designed is beautiful and would make for an incredible experience for visitors, immersing them in two times / worlds simultaneously through ghostly echos from the past.

Elia’s 3D imagery is also well researched, well thought out, beautifully illustrated, and depicts incredible detail.

A household scene, as depicted by Elia Marcacci, in in the Medieval Village once at Parco Archeologico di Tremona-Castello.
A household scene, as depicted by Elia Marcacci, in in the Medieval Village once at Parco Archeologico di Tremona-Castello.

Based on memory, the audio storytelling was also well done, to transport visitors through time and help them to better imagine life in the ancient village at Parco Archeologico di Tremona-Castello.

Current Challenges (as of 2019 Experience)

As Nadia mentioned, the Parco Archeologico di Tremona-Castello’s AR experience is not yet as she envisioned it in their concept video. In 2019, rather than seeing an active ghostly village around you, you can visit a number of 3D graphic illustrations of life in the village that are generally static, aside from minor movement, like a wisp of smoke. While these 3D illustrations are well done, when you are expecting to walk into an active ghostly village, this is disappointing.

For me, however, the most problematic part of this AR experience is the AR glasses. They are uncomfortable, are difficult to see through, especially if you yourself wear glasses, and are difficult to see the 3D scenes through. As such, visitors don’t get to fully appreciate how well Elia’s illustrations have been crafted.

Erica wearing AR glasses at Tremona Castello Archaeological Park

While the audio storytelling is well designed, the audio storytelling device is problematic, as it has to be set in the visitor’s centre and once you start it, it cannot be paused or replayed without returning to the visitor’s centre to have it reset. This means you can’t take the time to enjoy and immerse yourself in the Parco Archeologico di Tremona-Castello’s AR experience at your own pace, but must stick to the pace of the device. This also means that on a hot, sunny day, you can’t take breaks from the tour under the shade of a tree, but must stay in the sun, until the tour reaches its completion.

The AR glasses and audio guide also limit when visitors can enjoy the experience to the hours of the Visitor’s Centre.

Takeaways From This Extended Reality Experience

Nadia’s initial description of what she envisioned for Parco Archeologico di Tremona-Castello’s AR experience is similar to what I’ve been envisioning for the natural and cultural history augmented reality experiences that I’ve been dreaming of. It is exactly the experience that I was hoping for, and I suspect that with advancements in motion capture suits and world building becoming more accessible through game design engines, like Unreal Engine and Unity, it is becoming easier to make visions like this a reality in the future, making me excited for a future visit to Parco Archeologico di Tremona-Castello to see how the AR experience has evolved. It would seem that Elia has already constructed and made it possible to explore the outdoor spaces of the ancient village that once stood at Parco Archeologico di Tremona-Castello through his 3D animations. Having said that, I think tackling this in pieces of starting with specific scenes, rather than a whole ghostly village all at once, makes this much more manageable logistically and financially, as well as allowing for the opportunity to test drive and learn from the initial scenes before designing more.

The AR glasses were very problematic for me, however, and create an accessibility challenge and I imagine a large expense in establishing this experience. I personally would design a version of this that visitors could enjoy through their smartphones and tablets. With advances in smartphone and tablet technology, the concerns for visitors’ devices switching to an Italian telecommunications network, resulting in unexpected cellular charges, could be resolved in the following ways:

  • Creating a GPS triggered app for the AR experience, that visitors download before leaving the visitor’s centre. Visitors can then turn on airplane mode on their devices, and start the app while using the visitor’s centre wifi. A number of travel audio guide apps work this way, including the popular GuideAlong audio tours.
  • Create a WebAR version of the AR experience, and install a mobile wifi router (like a starlink mobile) at Parco Archeologico di Tremona-Castello for visitors to use while their devices are in airplane mode.

Being able to enjoy this AR experience through a smartphone or tablet would make it accessible to more people, would make Elia’s 3D illustrations easier to view and enjoy, and would allow people to enjoy the experience at their own pace, pausing and replaying it, if they wish. In this way, the AR experience could even be made accessible outside of the visitor’s centre’s hours, if so desired.

It should be noted that one advantage to AR glasses over WebAR is that the augmented realities resolution is usually better with AR glasses, but if the augmented reality is difficult to see through the AR glasses, then it doesn’t really matter how good the resolution is.

Food storage in the Medieval Village once at Parco Archeologico di Tremona-Castello, as depicted by Elia Marcacci.
Food storage in the Medieval Village once at Parco Archeologico di Tremona-Castello, as depicted by Elia Marcacci.

Summary of Takeaways:

  • creating a ghostly ancient village that can be viewed with augmented reality with ghostly figures actively taking part in village life is where the magic in this experience is for me
  • creating a series of ghostly scenes from the village is a more manageable starting point, and allows for the opportunity to test drive and learn from the initial scenes before designing more
  • AR glasses are uncomfortable for some and difficult for some people to see the augmented reality scenes in
  • AR and audio storytelling is more accessible when experienced through a smartphone or a tablet
  • concerns of smartphones or tablets switching to an Italian telecommunications network, could be resolved by having the AR experience accessible through a GPS triggered app or available through WebAR if a mobile wifi router was available for use at Parco Archeologico di Tremona-Castello for visitors to use while their devices are in airplane mode
  • being able to enjoy this AR experience through a smartphones or tablet would allow people to immerse themselves in the experience at their own pace, pausing and replaying it, if they wish

Future Building

In reflecting upon Parco Archeologico di Tremona-Castello’s AR experience, in addition to the desire to have this experience accessible through a smartphone or table, allowing me to pause and replay the audio storytelling, and to see the ancient village in ghostly form with active ghostly villagers, I’d also love to be able to interact with those scenes in some way. Some possible ideas for that:

  • the ability to take pictures with the ghostly village and villagers, including selfies
  • being able to listen in on the conversations of ghostly villagers
  • being able to learn an activity from the ghostly villagers, perhaps as they teach a ghostly child or apprentice
  • being able to partake in a festive event, by dancing or signing along with the ghostly villagers

The educator in me is also envisioning creating the village in VR for use in school classrooms, or possibly VR (or AR) teaser experiences that would get school kids and other visitors prepared for a visit with a bit of a background of the time period that they will be transported to and some questions or things to look out for and think about on their visit to Parco Archeologico di Tremona-Castello.

Exploring Tremona-Castello Archaeological Park with AR glasses.

What’s Your Take on this Augmented Reality Experience?

I’d love to hear your thoughts in comments below on what you think would make this a richer and more accessible AR experience, and if you have technical solutions for making this a more immersive and user friendly experience.

References

Hargreave, E., [StoryToGo]. (2023b, December 15). Tremona Castello Archaeological Park AR Experience [Video]. YouTube. https://www.youtube.com/watch?v=6f8NZaeC_do

La storia. (n.d.). https://aram-ti.ch/il-villaggio/la-storia/

Marcacci Elia. (2014, January 6). Ein Tag in Tremona-Castello. Medienarchiv. https://medienarchiv.zhdk.ch/entries/63853132-1225-498b-ba50-c0b38d3a9eb9

Martinelli, A. (2008). Tremona Castello. Dal V millennio a.C. al XIII secolo d.C. All’Insegna del Giglio.

MendrisiottoTurismo. (2016a, May 25). Il villaggio di Tremona-Castello in 3D [Video]. YouTube. https://www.youtube.com/watch?v=4gzp_A9ADrQ

Mendrisiotto Turismo. (2016b, August 10). Parco archeologico di Tremona-Castello, oggi [Video]. YouTube. https://www.youtube.com/watch?v=ri7HMuhSrVQ

MendrisiottoTurismo. (2016c, August 29). Anno Domini – Rievocazione storica al Parco archeologio di Tremona-Castello [Video]. YouTube. https://www.youtube.com/watch?v=hkCdTiknS-4

MendrisiottoTurismo. (2019a, May 15). Parco Archeologico di Tremona Castello [Video]. YouTube. https://www.youtube.com/watch?v=qHGKHO4bZBI

MendrisiottoTurismo. (2019b, July 12). Hunting for mysteries in time [Video]. YouTube. https://www.youtube.com/watch?v=Ty9LxvWVsOw

Mendrisiotto Turismo. (2023, May 19). La Regione da scoprire – Il Parco archeologico di Tremona 2023 [Video]. YouTube. https://www.youtube.com/watch?v=6BbQJVkru30

Parco Archeologico di Tremona-Castello (n.d.). Ticino Tourism. https://www.parco-archeologico.ch/en/

Società editrice del Corriere del Ticino SA. (n.d.). Tremona – TicinoTopTen. Ticino Top Ten. https://www.ticinotopten.ch/en/experiences/tremona