Back
Getting Started
Gestures
I integrated a familiar gesture recognized by the Vision Pro with a subtle modification, transforming it into the emblem for the "sign language mode." The hand gestures prominently symbolize sign language as well.
Clicking
ASL to Text
"Hey" in ASL
Activating ASL Communication with Siri
ASL Siri Integration
The Interactions
Setting up
ASL to text
Users will have the option to configure this implementation on 'settings,' enabling an extra 360-degree hand scan for improved recognition of sign language from various capture angles.
How it works
Using Vision-based sign language recognition (SLR) and training the sign language dataset via cloudKit makes text translation from sign language recognition achievable.
User Flow - Sign Language (ASL) to text
User Flow - Sign Language (ASL) to text
Communication
Be in the moment. Without restrictions.
Live-time ASL to text translation
Apple Vision Pro seamlessly integrates the SLR system into FaceTime and all Apple platforms, offering built-in speech-to-text functionality and now enabling ASL-to-text conversion.
Your emotions
come alive
Utilizing the implemented trained model capable of recognizing sign language and facial expressions, we introduce spatial implementations of the original FaceTime gestures, allowing for reactions to be triggered in real-time.
Productivity
Get on the same page. In the same space.
Accessibility | Siri
Siri understands ASL
By activating the sign language setting, Siri now comprehends and responds to sign language, eliminating the need for users to type out messages to converse.
Siri activated through "Hey Siri" trigger motion
Siri comprehending ASL for "What is the weather like today?"
Productivity enhanced with Siri
Directly drag and drop Siri messages and information directly into Apple applications like iMessage for swift productivity and seamless information sharing.
Design
Living in your space.
Heysu Oh
Last Updated : Nov 2024