Supporting conversations via mixed reality

A mixed reality system that supports conversation and memory recall, which intends to help aging adults connect with others and preserve memories in the moment.

A mixed reality system that supports conversation and memory recall, to help aging adults connect with others and preserve memories in the moment.

A mixed reality system that supports conversation and memory recall,  to help aging adults connect with others and preserve memories in the moment.

Role: research, interface design, video prototyping

Team: Irene Alvarado, Mackenzie Cherban, Julia Petrich, Rossa Kim

Duration: 4 Months (Spring 2017)

Sponsored by Microsoft Research

The Challenge

As a team of five designers, we were asked to design a product, service or solution that demonstrates the value and differentiation of Mixed Reality by Microsoft. For this project, we focused on cognitive decline. As the population ages, cognitive decline is and will continue to be a pressing issue. As designers, we, all, will have to take care not to ignore this issue.

The Solution

Moment is a mixed reality system that supports conversation and memory recall, which intends to help aging adults connect with others and preserve memories in the moment. It learns users' media and its context gradually and later suggests relevant media as memory prompts to help users to remember their past. It becomes part of users life and supports them before they experience a cognitive decline. Then it suggests and/or supervises actions if they need when they age.

Memory Sharing: Analog to Digital

From our research, we learned that speech is not always enough to tell our stories. On the other hand, physical memories like albums take time to curate. Mobile devices enabled us to create digital memories and changed how we find them back and share them. More services started to provide content for immersive storytelling and 3D memory preserving.

Searching Media Through Conversations

In the age of digital content, the media in our collections and on the internet is difficult to sift through. Often, there is too much out there.

Moment leverages searching through media. It works like a voice- assistant - but replacing the voice response with visual results.


How does Moment work?

A Machine That Learns From Conversations

In Moment, it’s not about the user conversing with the system. Instead, it relies on two users to have their natural conversation. As the system learns about the user, it facilitates a conversation with the user by pulling memory triggers (media) into the feed.


To determine any flat surface to be a workspace,  users can do "bloom" gesture or press the 'mic' button.


To scroll through the feed, user can gaze to the arrow or swipe a finger left to right on the remote.


To share a media, user can gaze on the item, tap & hold and drag it to the space by gesture or the remote.


To scale or rotate the item, user can hold the item and move hands vertical or horizontal direction. 


To delete the item, user can gaze on the item and swipe it from the boundries of the shared space.


Users can save their conversations as memory packets by pressing the save button on the controller.


Design Considerations

Isolating Private and Shared Spaces

To foster user autonomy and control, we built separate private and shared spaces into Moment. They allow users to decide what others can see while allowing for collaboration.


Leveraging Existing Practices of Sharing Media

We considered how users would onboard to the system using today’s tools so that Moment can begin with what is familiar. If users start to use it from now, it can help them to recall their past.

A browser extension or iOS app can help users to get started.

Designing for All in Mixed Reality

To be accessible to older adults, we explored how to maintain visual accessibility in Mixed Reality.

As we also believe that 'there is security in the tangible', we designed a remote controller. The grooves allow users to feel the direction they tracking on the pad, while mic button works as a tap-to-speak button.

To see how a physical remote would feel, we prototyped a tactile Hololens controller with Unity.


Low Fidelity

We developed and tested several prototypes. We started with simple paper mock-ups, and bodystorming to quickly test our ideas. We wireframed our interface with screen UX tools at first.

Among the process, we developed and tested several prototypes. We started with simple paper mock-ups, and bodystorming to quickly test our experience design ideas. After we decided on our interaction scenario, we wireframed our user interface in flat by using traditional UX tools at first.


Video Prototyping

We iterated our flat UI's on Cinema 4D to think more dimensional. Working with Cinema 4D for the first time was challenging, but it made us think beyond the screens. We designed our interface in Cinema4D, then compose it with real footage in After Effects.

First Iterations of Pinch & Drag, Scale, and Rotate Interactions

Native Hololens App

Native Hololens App

We also developed a Hololens app using Vuforia to show how our concept renders into the existing interaction patterns of Hololens.

Hololens Iterations of Pinch & Drag, Rotating, and Scaling Media

Going Forward

Learnings about Memory and Mixed Reality

With Moment, we learned the importance of user research and prototyping. We followed a user-centered research and design process while working with Mixed Reality. This helped us to see what works and what does not in our short time for prototyping. From our final prototypes, we learned:


Personal Learnings

Moment was a project that I was able to experience a rich user-centered design process and try different methods along the different stages. With Moment, I've seen not only the importance of the peer to peer communication but also working and communicating within a team.

As our project timeline was very strict and structured in terms of checkpoint deliverables, this made us start the design prototyping far later stages and relatively short period of time in the whole project duration. If I work on such a project in future, especially with a technology that I'm not familiar with, I'd definitely start prototyping earlier and use it as a research method.

From the technical perspective, while we are prototyping, we had some problems designing the user experience for Hololens. Some of these problems partly tie back to the availability of 'designer-friendly' developer documentation from Microsoft as it is not an end-consumer device. Others are due to our lack of technical capabilities to develop such interactions with tools like Unity etc. as five of us were designers. To be more successful in such a project, I think our teams should be more interdisciplinary.


Defining the Research Territory

We were interested in aging, which we saw as a diverse opportunity area. As technology evolves, we believe that we should design the world to be inclusive and accessible to this growing segment of the population.

Territory and Stakeholder Maps

We collected our ideas of what a human needs to age well into the five areas below. Then we have brainstormed around what opportunities these areas offer for Mixed Reality.

·  ·  · 

Exploratory Phase

We started our exploratory research with qualitative methods: a survey and postcards. The postcards asked Baby Boomers what their relationship with technology. We conducted interviews with experts, caregivers, and adults ranging from boomers to seniors.


Opportunity Spaces

As a result of our exploratory research. We have seen five gaps as opportunities for mixed reality (MR) to assist people as they age.



Along with opportunity areas, we also recognized three challenges surrounding our problem space in relation to mixed reality.

Can Mixed Reality be more than HMD? 
Current head-mounted display (HMD) technology is heavy, and in some ways, even makes a person temporarily disabled.

How can we reframe aging to be positive?
There are also many negative biases surrounding aging. They are difficult to solve as we try to make them one might look forward to.

How can we design technology that fails gracefully?
Boomers view technology as a tool. When something goes wrong, frustration ensues and trust in the technology diminishes.


Design Implications

We finalized our exploratory research with 7 design implications.


·  ·  · 

Generative Phase

During this phase, we devised a number of generative research methods. Along with a "magical device" workshop, we facilitated a collaging+card game workshop at a senior center. We iterated our collaging workshop for Boomers and brought it to them digitally.


Design Directions

Then we developed design directions that manifest our findings.

The lifelong companion: A product that grows as you age
The net: A safety mechanism for older adults aging in place
Living memories: Objects that contain the essence of memories
Share-it-forward: The knowledge transfer between generations


·  ·  · 

Evaluative Phase

In the last phase of our research, we brainstormed 16 different concepts, facilitated 4 hands-on technology workshops, 2 follow-up interviews, and designed 10 technology probes.


Speed Dating the Concepts

Through speed dating sixteen different concepts, we were able to see what resonated the most with Boomers. We have defined two bigger ideas as prototyping parallels to further explore.


Final Design Concept: Recall + Relate

In this design parallel, we focused on conversations and memory along with storytelling as our previous research suggested a non-negligible connection between cognitive decline and memories.


The origin of the "The Conversation Helper"

This parallel is most strongly related to our "the conversation helper" concept, which was an assistant and search service to support conversation and the recollection of memories.


Bodystorming and Prototyping the Idea

Parallel 2: Bodystorming and Prototyping

We acted out this idea to see how a conversation helper might have both in facilitating storytelling and aiding in memory recall.


What have we learned?

Parallel 2: What we learned?

We found ourselves thinking about many use cases and scenarios such as collaborative storytelling and embellishing stories for different audiences, and the values of such a system can offer.

This parallel is most strongly related to one of our speed dating concepts, the conversation helper, which is a smart assistant and search service to support conversation and the recollection of memories.


Iterating the Idea

Parallel 2: Selected Direction

As we go forward, we explored how the support and supervise modes can help a user to tell a personal story.


© 2018 Meriç Dağlı