🍔
Vision-Enhanced ASL: Promoting
Accessibility through VisionOS
Vision-Enhanced ASL: Promoting
Accessibility through VisionOS
Vision-Enhanced ASL: Promoting Accessibility through VisionOS
Accessibility enhancement for the Vision Pro platform. This initiative is geared towards individuals facing challenges with hearing impairment.
The concepts in this project are entirely hypothetical and are neither implemented by nor affiliated with Apple.
Accessibility enhancement for the Vision Pro platform. This initiative is geared towards individuals facing challenges with hearing impairment.
The concepts in this project are entirely hypothetical and are neither implemented by nor affiliated with Apple.
Vision-Enhanced ASL: Promoting Accessibility through VisionOS
Information
Information
Details
Details
Role
Role
Interaction Designer
Interaction Designer.
This was my individual thesis project during my fourth year at Carnegie Mellon University.
Interaction Designer.
Year
Year
Spring 2024
Spring 2024
Information
Role
Interaction Designer.
This was my individual thesis project during
my fourth year at Carnegie Mellon University.
Role
Spring 2024
Getting Started
Getting Started
Gestures
Gestures
Getting Started
I integrated a familiar gesture recognized by the Vision Pro with a subtle modification, transforming it into the emblem for the "sign language mode." The hand gestures prominently symbolize sign language as well.
I integrated a familiar gesture recognized by the Vision Pro with a subtle modification, transforming it into the emblem for the "sign language mode." The hand gestures prominently symbolize sign language as well.
I integrated a familiar gesture recognized by the Vision Pro with a subtle modification, transforming it into the emblem for the "sign language mode." The hand gestures prominently symbolize sign language as well.
Clicking
Clicking
Clicking
ASL to Text
ASL to Text
"Hey" Siri
"Hey" Siri
ASL Siri Integration
ASL Siri Integration
The Interactions
The Interactions
Users will have the option to configure this implementation on 'settings,' enabling an extra 360-degree hand scan for improved recognition of sign language from various capture angles.
Setting up ASL to Text
Setting up ASL to Text
Setting up ASL to Text
The Interactions
Users will have the option to configure this implementation on 'settings,' enabling an extra 360-degree hand scan for improved recognition of sign language from various capture angles.
Users will have the option to configure this implementation on 'settings,' enabling an extra 360-degree hand scan for improved recognition of sign language from various capture angles.
Users will have the option to configure this implementation on 'settings,' enabling an extra 360-degree hand scan for improved recognition of sign language from various capture angles.
Setting up ASL to Text
How it Works
Using Vision-based sign language recognition (SLR) and training the sign language dataset via cloudKit makes text translation from sign language recognition achievable.
ASL to Text
"Hey" Siri
ASL Siri Integration
Including Sign Language in settings menu
Vision Pro hand set up —> ASL hand set up
Including Sign Language in settings menu
Vision Pro hand set up —> ASL hand set up
Vision Pro hand set up —> ASL hand set up
Vision Pro hand set up —> ASL hand set up
How it works
Using Vision-based sign language recognition (SLR) and training the sign language dataset via cloudKit makes text translation from sign language recognition achievable.
View More +
How it works
Using Vision-based sign language recognition (SLR) and training the sign language dataset via cloudKit makes text translation from sign language recognition achievable.
View More +
How it works
Using Vision-based sign language recognition (SLR) and training the sign language dataset via cloudKit makes text translation from sign language recognition achievable.
Be in the moment. Without restrictions.
Be in the moment. Without restrictions.
Be in the moment. Without restrictions.
Communication
Communication
Apple Vision Pro seamlessly integrates the SLR system into FaceTime and all Apple platforms, offering built-in speech-to-text functionality and now enabling ASL-to-text conversion.
Live-time ASL to text translation
Live-time ASL to text translation
Live-time ASL to text translation
Communication
Apple Vision Pro seamlessly integrates the SLR system into FaceTime and all Apple platforms, offering built-in speech-to-text functionality and now enabling ASL-to-text conversion.
Apple Vision Pro seamlessly integrates the SLR system into FaceTime and all Apple platforms, offering built-in speech-to-text functionality and now enabling ASL-to-text conversion.
Apple Vision Pro seamlessly integrates the SLR system into FaceTime and all Apple platforms, offering built-in speech-to-text functionality and now enabling ASL-to-text conversion.
Live-time ASL to text translation
Live ASL to text on Facetime
Live ASL to text on Facetime
Utilizing the implemented trained model capable of recognizing sign language and facial expressions, we introduce spatial implementations of the original FaceTime gestures, allowing for reactions to be triggered in real-time.
Your emotions come alive
Your emotions come alive
Your emotions come alive
Utilizing the implemented trained model capable of recognizing sign language and facial expressions, we introduce spatial implementations of the original FaceTime gestures, allowing for reactions to be triggered in real-time.
Utilizing the implemented trained model capable of recognizing sign language and facial expressions, we introduce spatial implementations of the original FaceTime gestures, allowing for reactions to be triggered in real-time.
Utilizing the implemented trained model capable of recognizing sign language and facial expressions, we introduce spatial implementations of the original FaceTime gestures, allowing for reactions to be triggered in real-time.
Your emotions come alive
Facial recognition to trigger spatial gestures
Facial recognition to trigger spatial gestures
Get on the same page. In the same space.
Get on the same page. In the same space.
Get on the same page. In the same space.
Accessibility | Siri
Accessibility | Siri
By activating the sign language setting, Siri now comprehends and responds to sign language, eliminating the need for users to type out messages to converse.
Siri understands ASL
Siri understands ASL
Siri understands ASL
Accessibility | Siri
By activating the sign language setting, Siri now comprehends and responds to sign language, eliminating the need for users to type out messages to converse.
By activating the sign language setting, Siri now comprehends and responds to sign language, eliminating the need for users to type out messages to converse.
By activating the sign language setting, Siri now comprehends and responds to sign language, eliminating the need for users to type out messages to converse.
Siri understands ASL
Siri activated through "Hey Siri" trigger motion
Siri activated through "Hey Siri" trigger motion
Siri comprehending ASL for "What is the weather like today?"
Siri comprehending ASL for "What is the weather like today?"
Directly drag and drop Siri messages and information directly into Apple applications like iMessage for swift productivity and seamless information sharing.
Productivity enhanced with Siri
Productivity enhanced with Siri
Productivity enhanced with Siri
Directly drag and drop Siri messages and information directly into Apple applications like iMessage for swift productivity and seamless information sharing.
Directly drag and drop Siri messages and information directly into Apple applications like iMessage for swift productivity and seamless information sharing.
Directly drag and drop Siri messages and information directly into Apple applications like iMessage for swift productivity and seamless information sharing.
Productivity enhanced with Siri
Cross platform interaction : Siri, iMessage, Weather
Cross platform interaction : Siri, iMessage, Weather
Design
Design
Living in your space.
Get on the same page. In the same space.
Heysu Oh
Heysu Oh
Last Updated : Nov 2024
Last Updated : Nov 2024
Heysu Oh
Last Updated : Nov 2024