Vision-Enhanced ASL: Promoting Accessibility through VisionOS
Promoting Accessibility through VisionOS
Accessibility enhancement for the Vision Pro platform. This initiative is geared towards individuals facing challenges with hearing impairment.
Accessibility enhancement for the Vision Pro platform. This initiative is geared towards individuals facing challenges with hearing impairment.
Accessibility enhancement for the Vision Pro platform. This initiative is geared towards individuals facing challenges with hearing impairment.
Challenge
My final thesis capstone design project.
My final thesis capstone design project.
Tools
Ae, Figma, Bezi
Ae, Figma, Bezi
Duration
Spring 2024 - 12 weeks
Spring 2024 - 12 weeks
*The ideas within the project are purely conceptual and have not been actually implemented by Apple.
*The ideas within the project are purely conceptual and have not been actually implemented by Apple.
Getting Started
Gestures
I integrated a familiar gesture recognized by the Vision Pro with a subtle modification, transforming it into the emblem for the "sign language mode." The hand gestures prominently symbolize sign language as well.
User Flow
Sign Language (ASL) to text
The Interactions
How it works | Sign Language Recognition
Using Vision-based sign language recognition (SLR) and training the sign language dataset via cloudKit makes text translation from sign language recognition achievable.
Communication
Be in the moment.
Without restrictions.
Productivity
Get on the same page.In the same space.
Design
Living in your space
The Interactions
The Interactions
Setting up ASL to text
Setting up ASL to text
Users will have the option to configure this implementation on 'settings,' enabling an extra 360-degree hand scan for improved recognition of sign language from various capture angles.
Users will have the option to configure this implementation on 'settings,' enabling an extra 360-degree hand scan for improved recognition of sign language from various capture angles.
Accessibility | Siri
Siri understands ASL
By activating the sign language setting, Siri now comprehends and responds to sign language, eliminating the need for users to type out messages to converse.
User Flow
Sign Language (ASL) to text
Getting Started
Gestures
I integrated a familiar gesture recognized by the Vision Pro with a subtle modification, transforming it into the emblem for the "sign language mode." The hand gestures prominently symbolize sign language as well.
The Interactions
How it works | Sign Language Recognition
Using Vision-based sign language recognition (SLR) and training the sign language dataset via cloudKit makes text translation from sign language recognition achievable.
Communication
Be in the moment.
Without restrictions.
Live-time ASL to text translation
Live-time ASL to text translation
Apple Vision Pro seamlessly integrates the SLR system into FaceTime and all Apple platforms, offering built-in speech-to-text functionality and now enabling ASL-to-text conversion.
Apple Vision Pro seamlessly integrates the SLR system into FaceTime and all Apple platforms, offering built-in speech-to-text functionality and now enabling ASL-to-text conversion.
Your emotions come alive
Utilizing the implemented trained model capable of recognizing sign language and facial expressions, we introduce spatial implementations of the original FaceTime gestures, allowing for reactions to be triggered in real-time.
Productivity enhanced with Siri.
Directly drag and drop Siri messages and information directly into Apple applications like iMessage for swift productivity and seamless information sharing.
Siri activated through "Hey Siri" trigger motion
Siri comprehending ASL for "What is the weather like today?"
Your emotions come alive
Utilizing the implemented trained model capable of recognizing sign language and facial expressions, we introduce spatial implementations of the original FaceTime gestures, allowing for reactions to be triggered in real-time.
Productivity
Get on the same page.
In the same space.
Accessibility | Siri
Siri understands ASL.
By activating the sign language setting, Siri now comprehends and responds to sign language, eliminating the need for users to type out messages to converse.
Siri activated through "Hey Siri" trigger motion
Siri comprehending ASL for "What is the weather like today?"
Productivity enhanced with Siri.
Directly drag and drop Siri messages and information directly into Apple applications like iMessage for swift productivity and seamless information sharing.
Design
Living in your space.
Hey.