2025 · Lead Developer / Researcher
DuoSign
Text-to-Sign Language Translation
- Discipline
- ResearchMachine learningCoding
- For
- Study
- Stack
- ReactPythonMediaPipeThree.jsNLPComputer Vision3D RenderingInverse KinematicsASLAccessibility
Problem
Deaf and hearing people still lack quick, everyday translation between English and American Sign Language.
What I built
A React + Three.js signing avatar driven by a text-to-gloss NLP pipeline, plus live sign recognition with MediaPipe.
Result
57% average accuracy across 200+ ASL classes on WLASL.
Built a full-stack accessibility app achieving 57% accuracy across 200+ ASL classes on the WLASL dataset
What went into it
- Engineered a full-stack accessibility application that converts English text to American Sign Language (ASL) avatar animations and recognises live hand signs via MediaPipe.
- Built the frontend in React with Three.js 3D avatar rendering and inverse kinematics, creating realistic sign language animations from text input.
- Integrated a custom text-to-gloss NLP pipeline on the backend, translating natural English sentences into sign language gloss notation for avatar interpretation.
- Achieved an average of 57% accuracy across 200+ ASL classes on the WLASL dataset, demonstrating viable real-world sign language translation capabilities.
How I thought about it
I split the problem in two: understanding the sentence (text-to-gloss) and performing it (avatar + inverse kinematics). Keeping the gloss layer explicit made the avatar debuggable, at the cost of some natural phrasing.