How might we leverage augmented reality to help athletes practice gameplays more effectively?
timeline
10 weeks
Jan - Mar 2018
team
Nicolas Germann
Kate Johnson
Daphne Liang
Laura Lopez
Luke McJunkin
involvement highlights
Interaction Flow
Visual Design
High-fidelity Prototyping
Unity/ARKit Development
Holographic Interaction Experience
Augmented Reality (AR) has been a game-changer for interaction design due to its 3-dimensional, immersive, yet realistic nature. We were tasked with designing a holographic interaction experience for the Microsoft HoloLens, with a focus on producing a high-fidelity AR prototype.
Opportunities in AR Sports Training
From our field observation and competitive analysis, we identified sports training as a potential use case for AR since it’s crucial for the interactions to be immersive without being disruptive to or disconnected from the real world.
Introducing ARena
A multimodal application suite for sports strategy communication and simulation using Augmented Reality and Over-the-Air deployment. The coach will be using a tablet, while the athlete will be using a HoloLens.
Choosing and Deploying Plays
The coach can add, edit, and ultimately choose which strategy (play) to be deployed to the athlete's HoloLens for practice using a flick up motion.
Viewing Stats
ARena also collects individual athlete's practice statistics (completion time, accuracy, drills completed) for easy viewing, managing and comparison.
Visualizing and Practicing Plays
The play visualization is then procedurally generated and anchored in space by the athlete's headset. The athlete can clearly visualize the intended plays, follow the highlighted trail to practice movement path, and observe virtual teammates' positions.
Pause
Resume
Rewind/Fast Forward
Gestural Controls
Using HoloLens' gesture recognition, the athlete can fully control the experience to better comprehend the strategy.
—
Multimodality for Multiple Stakeholders
From our secondary research, we decided that multimodal experience would be a good fit for ARena because of its distinct groups of stakeholders (coach and athlete) with varied level of exposure to technology and involvement in the physical activities. Inspired by Apple’s AirDrop, we designed the over-the-air deployment concept for seamless interactions between the coach and athlete.
Interaction Flow
From the initial concept, we moved forward with fleshing out a comprehensive interaction flow to identify necessary screens or scenes design.
AR Prototyping in Unity
Our initial introduction to AR prototyping started with building the scene in Unity. Using simple 3D geometries and standard assets pack, we built out the initial setup for our experience.
Previsualization
To calibrate object and UI positioning in 3D space, I then implemented a first-person camera for rapid prototyping before porting to HoloLens.
Our initial athlete's UI implementation featured a static and fixed on screen display for play-related information
Shipping to the HoloLens
Shipping to the HoloLens enabled us to fully experience the initial build in augmented reality. But more importantly, the HoloLens build also uncovered various usability issues and technical constraints that needed addressing:
Eye-level and fixed on glass interfaces were obstructive to training activities.
Due to studio policy, we could not take the HoloLens outdoor for further testing.
Developing mainly on MacOS, my workflow was substantially affected by the HoloLens’ Windows environment requirement.
Porting to iPhone
As a workaround to technical and policy constraints, I ported the prototype to the iPhone using ARKit and XCode. I then iterated and made necessary changes to the interface, i.e. locking it in 3D space, to minimize distraction while still providing necessary information.
This final prototype enabled us to experience true augmented reality on the field but without gesture recognition capabilities. However, making the necessary trade-off allowed us to move fast and focus on the main experience.
—
Future Steps
expand beyond football
The same model of strategy communication and visualization in AR has potentials to be implemented in other team and individual sports, e.g. basketball, baseball, motorsport, etc.
support multiple athletes simultaneously
An essential part of team practice is, well, teamwork. To best support simultaneous team training, we have to think about possible athlete-to-athlete and coach-to-athletes interactions.
investigate hardware capability
ARena was designed for a near future where AR technology is more refined and portable. As we move on to further iterate ARena, we want to investigate deeper into what hardware capabilities and options are being developed.
Takeaways
deal with constraints
Despite the various constraints presented in this project, I was excited to be adaptive and always explore alternatives in tools and platforms to push the concept to its highest-possible fidelity.
understand limitations beforehand
As AR is still a new technology, we had to intentionally make assumptions about the technology to complete the project in time. In retrospect, I would have dedicated more time to fully understand the technical limitations before starting.
2D screen vs. 3D scene
Prototyping for AR presented a variety of elements that I hadn't considered before when designing for 2D scenes. Scale, angle, distance, and line of sight all matter in making the AR experience a delightful one.