Clicky

moment_splash

Moment

Moment

Supporting conversations through mixed reality

Supporting conversations for memory recall

A mixed reality system that supports conversation and memory recall, which intends to help aging adults connect with others and preserve memories in the moment.

A mixed reality system that supports conversation and memory recall to help aging adults connect with others and preserve memories in the moment.

A mixed reality system that supports conversation and memory recall, to help aging adults connect with others and preserve memories in the moment.

A mixed reality system that supports conversation and memory recall,  to help aging adults connect with others and preserve memories in the moment.

Role: research, UI/UX, video prototyping

Team: Julia Petrich, Irene Alvarado, Mackenzie Cherban, Rossa Kim

Brief: Design a product, service or solution that demonstrates the value and differentiation of Mixed Reality.

Sponsored by Microsoft Research in Spring, 2017

Moment learns users' media and its context gradually, and later suggest relevant media as memory prompts to help users to remember their past in mixed reality. It becomes part of users life, and support them before they experience a cognitive decline. Then suggest, and/or supervise actions if they experience any memory loss when they age.

A Pressing Issue: Cognitive Decline

So why look at cognitive decline? As the population ages, cognitive decline is and will continue to be a pressing issue. This is something that we, all, will have to take care not to ignore.

territory-aging-numbers1
Defining the Quality of Life

Existing tech solutions for older adults focus on the physiological and safety needs. The quality of life is defined by the growth-related needs. The personal growth does not stop in later years of life.

From our semester-long research, we believe that a mixed reality solution can interact with older adults beyond traditional definitions.

final-maslov-11
Memory Sharing: Analog to Digital

Speech is not always enough to tell our stories. On the other hand, physical memories like albums take time to curate. Mobile devices enabled us to create digital memories and changed how we find them back and share them. More services started to provide content for immersive storytelling and 3D memory preserving.

Searching Media Through Conversations

Media in our collections and on the internet is difficult to sift through. In essence, there is often too much out there.

Moment leverages searching through media. It works like a voice- assistant - but replacing the voice response with visual results.

why-41
How does Moment work?

A Machine That Learns From Conversations

In Moment, it’s not about the user conversing with the system. Instead, it’s reliant on two users to have their natural conversation. As the system learns about the user, it facilitates a conversation with the user by pulling memory triggers (media) into the private feed.

userflow_21
Design Considerations

Isolating Private and Share Places

To foster user autonomy and control, we built separate private and shared spaces into Moment. They allow users to be in the driver’s seat, deciding what others can see, while allowing for collaboration.

autonomy

Leveraging Existing Practices of Sharing Media

We considered how users would onboard to the system using today’s tools. This way, Moment can begin with what is familiar. If users start to use it from now, Moment can help them to recall their past.

A browser extension or iOS app can help users to get started.

Designing for All in Mixed Reality

We explored how to maintain visual accessibility in Mixed Reality, with our tests on the Hololens.

Also, we believe that 'there is security in the tangible'. The Grooves on our remote allow users to feel the direction they tracking on the pad, while microphone button works as a tap-to-speak button.

We also prototyped how a physical remote would feel. We connected a Particle dev-board with a tactile button to Unity, and then Hololens.

Prototyping

Low Fidelity

Low Fidelity

We developed and tested several prototypes. We started with simple paper mock-ups, and bodystorming to quickly test our ideas. We wireframed our interface in flat by using traditional UX tools at first.

Among the process, we developed and tested several prototypes. We started with simple paper mock-ups, and bodystorming to quickly test our experience design ideas. After we decided on our interaction scenario, we wireframed our user interface in flat by using traditional UX tools at first.

Medium Fidelity

We iterated our flat UI's on Cinema 4D to think more dimensional. Working with Cinema 4D for the first time was challenging, but it made us think beyond the screens. We designed our interactions in Cinema4D in 3D, then compose it with real footage in After Effects.

First Iterations of Pinch & Drag, Scale, and Rotate Interactions

Video Prototyping

We video-sketched with real-life footage to show the interactions.

Current Iterations of Pinch & Drag, Scale, Rotate, Delete, Save, and Spatial Mapping Interactions

Native Hololens App

Native Hololens App

Native Hololens App

Finally, we developed a native Hololens app to show how our design can be transferred into the existing interaction patterns of Hololens.

Hololens Iterations of Pinch & Drag, Rotating, and Scaling Media
What have we learned from Moment?
What have we learned from Moment?
whatwelearned

·  ·  · 

Process

Defining the Research Territory

We were interested in aging, which we saw as an incredibly diverse experience at the intersection of all sorts of disabilities. As technology continues to evolve, we believe that we should design the world to be inclusive and accessible to this growing segment of the population.

Territory and Stakeholder Maps

Starting from people, we collected our ideas of what a human needs to age well into the five areas below. Then we have brainstormed around what opportunities these areas offer for Mixed Reality.

·  ·  · 

Exploratory Phase

We started our exploratory research with qualitative methods: a survey, and postcards. The postcards asked Baby Boomers what their relationship with technology. We also conducted interviews with experts, caregivers, and adults ranging from baby boomers to seniors.

exp_res_numbers

Opportunity Spaces

As a result of our exploratory research. We have seen five gaps as opportunities for mixed reality (MR) to assist people as they age.

exp_opp_spaces2

Challenges

Along with opportunity areas, we also acknowledge three challenges surrounding our problem space in relation to mixed reality.

Can Mixed Reality be more than HMD? 
Current head-mounted display (HMD) technology is heavy, and in some ways, even makes a person temporarily disabled.

How can we reframe aging to be positive?
There are also many negative biases surrounding aging. They are difficult to solve as we try to make them one might look forward to.

How can we design technology that fails gracefully?
Boomers view technology as a tool. When something goes wrong, frustration ensues and trust in the technology diminishes.

exploratory-challenges1

Design Implications

As a final step in our exploratory phase, we defined 7 design implications.

generative_res-design-imp-upd1
exp_design_implications_2

·  ·  · 

Generative Phase

During this phase, we devised a number of generative research methods. Along with a "magical device" workshop, we facilitated a collaging+card game workshop at a senior center. We also reframed our collaging workshop for Boomers and brought it to them digitally.

generative_res-numbers

Design Directions

Then we developed design directions that manifest our findings.

The lifelong companion: A product that grows as you age
The net: A safety mechanism for older adults aging in place
Living memories: Objects that contain the essence of memories
Share-it-forward: The knowledge transfer between generations

generative_res-design-dir-recap1

·  ·  · 

Evaluative Phase

In the last phase of our research, we brainstormed 16 different concepts, facilitated 4 hands-on technology workshops, 2 follow-up interviews, and designed 10 technology probes.

evaluative-numbers

A System That Evolves

We designed a system that provides a clear value now, then changes over time as it becomes more critical and may take control more.

evaluative-parallel2-system1

Speed Dating the Concepts

Through speed dating sixteen different concepts, we were able to see what resonated the most with Boomers. We have defined two bigger ideas as prototyping parallels to further explore.

For our final design solution, we went with the parallel Recall + Rebate, which we believed that it can help with the cognitive decline of more users than our other parallel. 

evaluative-system-16-concepts

Final Design Concept: Recall + Relate

Final Design Concept: Recall + Relate

In this design parallel, we focused on conversations and memory along with storytelling as our previous research suggested a non-negligible connection between cognitive decline and memories.

evaluative-parallel2-system1

The origin of the "The Conversation Helper" Concept

This parallel is most strongly related to one of our speed dating concepts, the conversation helper, which was an assistant and search service to support conversation and the recollection of memories.

evaluative-parallel2-concept1

Bodystorming and Prototyping the Idea

Parallel 2: Bodystorming and Prototyping

We acted out this idea to see how a conversation helper might have both in facilitating storytelling and aiding in memory recall.

2017-09-16-15_14_20

What have we learnt?

What have we learnt?
Parallel 2: What we learned?

We found ourselves thinking about different use cases and scenarios such as collaborative storytelling and embellishing stories for different audiences, and the values of such a system can offer.

This parallel is most strongly related to one of our speed dating concepts, the conversation helper, which is a smart assistant and search service to support conversation and the recollection of memories.

evaluative-parallel2-result1

Iterating the Idea

Parallel 2: Selected Direction

As we go forward, we continued to prototype ways that conversation helper could come to life in two of the stage. We explored how the support and supervise modes can help a user to tell a personal story.

2
3

© 2017 Meriç Dağlı