Real-Time Sign Language Translation with MediaPipe, Flutter, and Gemini Nano
SignSpeak is a mobile application that enables real-time translation of American Sign Language (ASL) and custom hand gestures into natural English sentences using on-device machine learning. It leverages MediaPipe for hand landmark detection and Gemini Nano for on-device language processing, ensuring full offline functionality and user privacy. The app supports customizable vocabulary, allowing users to add new signs or phrases without retraining the model.
- ▪SignSpeak uses MediaPipe to detect 21 hand landmarks per hand, capturing detailed 3D hand poses for gesture recognition.
- ▪The app normalizes hand coordinates relative to the wrist and middle finger span to ensure consistent input regardless of hand position or distance from the camera.
- ▪Gemini Nano enables on-device natural language generation, allowing translation without internet connectivity or data transmission.
- ▪Feature extraction must precisely match between training and inference, including the use of Euclidean distance for normalization, to maintain model accuracy.
- ▪Users can customize the app’s vocabulary by adding new signs, names, or phrase shortcuts without requiring model retraining.
Opening excerpt (first ~120 words) tap to expand
try { if(localStorage) { let currentUser = localStorage.getItem('current_user'); if (currentUser) { currentUser = JSON.parse(currentUser); if (currentUser.id === 3924614) { document.getElementById('article-show-container').classList.add('current-user-is-article-author'); } } } } catch (e) { console.error(e); } noor yadallee Posted on May 16 Real-Time Sign Language Translation with MediaPipe, Flutter, and Gemini Nano #machinelearning #gemini #mediapipe #flutter SignSpeak Real-Time Sign Language Translation with MediaPipe, Flutter, and Gemini Nano Build with AI - 2026 | Noor SignSpeak is a mobile application designed to bridge the communication gap for the Deaf and Hard-of-Hearing community.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at DEV.to (Top).
Discussion
0 commentsMore from DEV.to (Top)
