Phipps XR Interaction

Mixed Reality - Fall 2019
Duration: 3 Weeks
Tools: Microsoft Hololens, Oculus (Gravity Sketch), Blender, Xcode Reality Composer, After Effects
Collaborators: N/A

Role: Conducting user research, storyboarding, and prototyping in Augmented Reality and Virtual Reality spaces.


Phipps Conservatory is a botanical garden and historical landmark in Pittsburgh, Pennsylvania. The conservatory showcases hundreds of plants from around the world, including Cuba, Canada, and the States. Currently, visitors can explore the space by themselves or on guided group tours.

As technology and design mediums advance, so should the way we visually and cognitively engage with information. In this project, I explored how XR interactions can enrich the way individuals both experience and learn about natural ecosystems.

01 / Problem Space


The majority of plant life at the Phipps Conservatory is curated and installed in an inevitable, showcase-like manner. Plants are taken out of their contextual environments and as a result, visitors are unable to gain a holistic understanding of the exhibited plant's role in its natural ecosystem. Given these concerns, I framed my problems through these two questions:

“How can I provide users with a better contextual understanding of exhibition pieces and their natural environments and in the process, bring excitement to learning through technological innovation?
“How can I encourage visitors with a call to action in regards to their role as a player in their own communities?”


A tabletop AR terrain that adapts depending on the user's placement of physical action trophies. This design solution seeks to help visitors visualize an exhibited plant in its broader natural habitat, as well as the role the Anthropocene plays in these wildlife environments.

02 / Visiting Phipps

I visited Phipps in hopes of getting a better understanding of what it’s actually like to walk through the environment. First, I observed that Phipps is broken up into several theme-based showcase rooms, an example being the temporary Christmas decorations and model train system, as well as their permanent Cuba exhibit. I also found that Phipps provided interactive environments for users to engage with. Many of these involved things such as maps and microscopes.

Various Phipps exhibition halls

03 / Stakeholders

I visited Phipps not only to better understand the exhibition material itself, but also the types of people that visited. From my time there, I realized that I could separate my target users into primary and secondary users depending on factors such as their duration in Pittsburgh, age, and intent for visiting. The primary users are everyday, kids, college students, and adults. The secondary users would be people thinking of investing in Phipps - this includes sponsors, donors, and individuals interested in holding a venue at Phipps.

From there, I created a series of relevant user archetypes. The personas below visualize a variety of potential stakeholders, ranging from 8th grades, engineer dads, to retired art collectors. By creating these personas, I realized that to be able to create a solution that caters to the wide range of stakeholders, I had to successfully entertain, educate, and engage kids, adults, and potential clients

04 / Ideation

Breaking down my stakeholders helped me realize a few things. To truly satisfy both my primary and secondary users, I would need to create a highly intuitive experience capable of engaging, entertaining, and educating various groups of individuals that all have diverse motives.

Throughout this process, I framed my thinking around how I can best educate visitor about a plant's natural environment. Doodles turned into ideas and eventually I saw myself begin conceptualizing a hybrid tabletop experience.

05 / Prototyping

Physical Prototyping: Text-based Information

A tricky part of prototyping for spatial interfaces is communicating the way information is displayed. With time as a primary factor, I built physical mockups of information cards.

VR Prototyping: Terrain Projection

From there, I took my initial tabletop design ideas into Oculus' Gravity Sketch and began modeling in 3D space. Doing so allowed me to design a one-to-one model, giving me a more realistic understanding of what it might be like in world space.

AR Prototyping - Terrain Projection

Designing in VR proved to be extremely valuable in many regards, but it couldn't mimic the qualities of world space. I transitioned out of the headset into Xcode's Reality Composer so that I could prototype interactively with AR. Below are a few iterations.

One of the biggest frustrations for me was AR occlusion. Since my design concept relied on heavy engagement of physical objects on top of digital projections, it was really hard to compellingly communicate my speculative design. Upon collaboration with peers and mentors, I realized part of this problem can be solved by replacing the digital landscape with a physical one.

06 / Final Interaction

Action Trophies

To go more in-depth about the physical aspect of this interaction, every AR tabletop within Phipps will have a “Start” trophy, that is the trophy that begins the entire simulation. When the user moves the trophy to certain spots on the table, the terrain will react accordingly depending on what type of action trophy it is. In this case, I have included cattle herding and oil fracking. The goal of this is for users to understand how human actions can completely alter the way a natural ecosystem continues to exist.

Final Prototype

At the end of this process, I developed this speculative concept video that visualizes how an individual might engage with this interaction.

Multiple locations in Phipps

Since each plant species has its own unique environment, it’s only fitting that there are several distinguished tabletop platforms spread across the entire botanical garden.

07 / Conclusion

In summary, my goal for this design challenge was to enhance the way people engage in the Phipps Conservatory through some XR medium. As someone who just recently began prototyping in spatial environments, this project taught me a lot about the affordances of augmented reality tools and the limitations of how certain ideas and be communicated. I’ve come to realize that these restraints are quite logical sense I’m conceptualizing future-based interactions with available tools in the present.

In the near future, I would love to explore prototyping tools such as Unity and Blender; I believe that the combination of these two tools will elevate my ideas to a new level, allowing me to solve bigger scaled issues in the man-made world.