AR for Healthcare

Making physical rehabilitation more engaging for children patients

Why Gamification?

Why Augmented Reality?

Therefore, we set our design goal as to leverage AR to gamify the rehabilitation experience to be more engaging for children patients.

User Research

After several meetings with physicians and nurses from the UCSF Children’s hospital, we understand that the gamified experience is applicable to patients who go through occupational therapy, obesity or diabetes exercise, or at-home physical rehabilitation training. It is not applicable for patients who suffered a traumatic brain injury and severe mobility inability. Facing a user group with special needs, the game needs to ensure a safe playground.

(view more research results in this term paper)

Game Research

Both “Tracking” and “Board Game“ are good choices. The “Board Game” has a high level of repeatability because of the randomness and potential multi-player mode, which can reduce loneliness and motivate children to play repeatedly. The “Tracking Game” is also a good fit for exercise because there is a model for users to follow. The model helps to standardize the game duration and the user movements, therefore can gather useful exercise data, such as calories, steps, etc., to share with the therapists and parents.

Game Structure and Elements

We decided to create a tracking game with a pet raising theme and mini exercise games. The minigames included exercises with the lower body, upper body, and positive facial expressions. Children raise and interact with a virtual pet, and can decorate their environment allowing for creativity and customization. To get more professional ideas, we sought feedback from Professor D'Esposito from the Department of Psychology at UC Berkeley on the cognitive effects. We also had meetings with a game designer from Electronic Art to provide suggestions. Consolidating all opinions, we built the rough game diagram.

Prototypes and Usability Testing

As all of us are new to AR prototyping, we first used 2D prototyping methods(e.g Figma, Hand-drawing) and then moved to 3D prototyping methods(e.g Reality Composer) for usability testing. This “separation“ approach turns out to be inefficient and problematic. As we conduct more usability testings, we are also iterating prototyping methods.

1. Main Interaction

Problem: Movement-based interactions create inconsistency between visual and sensory feedback

In the first attempt, we expect children to complete different exercise tasks based on the blocks they step on through an AR broad game. However, after we built prototypes in Reality Composer. The prototypes show problems.

We invited 5 college students for usability testing. One said:

“After getting ready to step on a tile, I actually stepped on the flat ground. “

There is an inconsistency between expected and actual sensory feedback. Imprecision of AR localization and instability of AR object rendering also makes it hard to detect the collision between the user and the AR object. They together can lead to safety concerns. This also leads to another question: how can we quickly create and iterate AR prototypes for Usability Testing?

Solution:

Balancing movement-based interaction with traditional “Tab, Drag & Drop”

In the second time, we tried to balance the traditional UI interactions such as tabbing and dragging with the more AR-specific interactions. Children trigger games by tabbing on the button instead of walking on certain blocks. To encourage interaction with the environment and people and create a customizable experience, we incorporate AR Decoration into the game. By clicking on “Decorating Room”, children enter the editing mode. They are able to drag and drop 3d assets.

Since we want to maximize players’ visibility, we prototyped with different visual representations of 3D assets placement. We decided to display shape instead of the actual asset while players decide where to place the asset to avoid blocking the views.

2. Exercise type

Problem:

Exercise requiring both upper and lower body movement can be dangerous

For the exercise game, we tried multiple interaction/AR-compatible exercising models. We combined Figma prototypes and quick Reality Composer demos to test our ideas. With these prototypes, we quickly found that movements associated with both the upper and lower body, especially those that post time constraints can lead to rushed movements and unstable visions. For example, catching fruits before it falls to the ground or aiming at monsters who disappear in 5 seconds.

Solution:

Motion detection for one direction

We used built-in sensors to analyze the users’ movement trajectories and incorporated animated models into the movement in order to make the game more interactive. In this game, children follow their pets to squat and pick up the treasures.

3. Reward System

We want to create a motivating and non-addictive reward system. Similarly, we thought about using more UI instructions vs tutorials for the mini-game. Due to the concerns of visibility and intuitiveness, we decided to create a first-time tutorial for children to follow their dragon pet. The interactions only include clicking and moving around. We enlarged the fonts and AR objects so it’s easier for children to see while in motion. We also included more positive suggestions such as “Perfect!“, “Almost there, try again!“ to encourage the children. After the game, we would show a Game Summary of successful squats and rewards.

4. Player Feedback

In the last game, we expect children to mimic the pet’s facial expressions and head movement(smiling, nodding, shaking head) However, we faced a question: do we want the users to see their own self-reflection? We proposed 2 hypotheses:

1. Children see pet’s facial expressions, followed by their own reflections in the form of

   1. Camara view

   2. Filtered view (eg. Snapchat filter)

   3. Cartoon figure view (eg. iPhone’s Memoji) 👍

2. Children only see the pet’s facial expressions

I did A/B testing in a graduate Mixed Reality Computer Science class (CS294-Immersive Computing and Virtual Reality). More than 50% of students think that cartoon reflections not only provide feedbacks so users know where they did wrong but also avoid the problem of privacy concerns and limited representations. I recorded four video pieces of screen view and a 3rd person view while playing the game to make the prototypes for A/B testing as realistic as possible.

Final Prototypes

Decoration(Figma)

Children are able to project images onto the real world by decorating virtual objects. We want children to position without blocking, therefore we used 2D shapes instead of 3D assets to indicate location.

Squat Game (Reality Composer + Video prototyping)

Facial Expression Game (Figma)

In the last game, we expect children to mimic the pet’s facial expressions and head movement for a potisive cognitive effect. From the previous research, we decided to use Animal Avatar because it

1. Provide feedback for performance evaluation

2. Avoid unsatisfied self-imagery and limited representations.

Localization (Development)

When users save the virtual decoration in their environments, we use the ARKit’ built-in function to save the anchor of the virtual objects. The next time when they open it, it will use image-based matching to restore the world coordinates and display the object using its anchor.

Game center for guardians

Future Impact Measurement

Due to the quarantine, the project was paused in Jan 2021 when the game was only partially developed. In the ideal case, we want to use the following defined success metrics with the targeted user: children going through physical rehabilitation and their guardians(families and physicians).

Drawing: Yuhan Xie 

I hope this project sparks new insights for AR applications in the healthcare field, which I believe is valuable as it provides tangible help to the vulnerable group across demographics. I sincerely look forward to the team progress and the potential influence it will have!

👋 If you are interested in AR projects, here’s my previous project on an AR journey that helps children to navigate in a science museum.