ISL Connect is a real-time assistive AI application that converts spoken language into Indian Sign Language (ISL) visual outputs. The system captures live speech, converts it into text using speech recognition models, processes it using NLP techniques, and maps the processed text to corresponding ISL visual representations (GIFs/images).
The system aims to reduce the communication gap between hearing individuals and the deaf community by enabling natural speech-to-sign translation.
Sign language is the primary language used by deaf individuals. However, most hearing individuals are not trained in sign language, creating a significant communication barrier.
Traditional communication requires:
- Manual interpreters
- Time-consuming sign language learning
- Limited accessibility in daily scenarios
ISL Connect bridges this gap by enabling real-time speech-to-sign translation, allowing seamless interaction between hearing and deaf individuals.
• Real-time speech recognition using microphone input
• Speech-to-text pipeline using Google Speech API and Sphinx (offline fallback)
• NLP-based preprocessing for text normalization and phrase detection
• Dictionary-based machine translation mapping text phrases to ISL gestures
• Visual output generation using ISL GIFs and images
• Lightweight desktop interface using EasyGUI
• Noise calibration and energy thresholding for robust speech detection
Speech Input (Microphone) │ ▼ Speech Recognition Engine (Google Speech API / Sphinx Offline) │ ▼ Text Preprocessing (NLP) • Lowercasing • Tokenization • Phrase normalization │ ▼ Dictionary-Based Translation (Text → ISL mapping) │ ▼ Visual Renderer (ISL GIF/Image Output)
- Initialize system
- Calibrate microphone energy threshold to detect ambient noise
- Capture speech input via microphone
- Convert speech to text using speech recognition
- Normalize and preprocess the text
- Check if phrase exists in predefined ISL dictionary
- If phrase exists → display corresponding ISL GIF
- If phrase not found → break phrase into letters and display sequential gestures
- Continue listening until user says "goodbye"
- Exit application
| Metric | Value |
|---|---|
| Supported ISL phrases | 120+ |
| Average speech recognition latency | ~0.8 sec |
| Translation pipeline latency | <1.2 sec |
| End-to-end response time | ~2 sec |
| Speech recognition accuracy | ~91% (clean audio) |
| Offline fallback accuracy | ~84% |
The system was tested across 50+ voice samples with varying accents and background noise conditions.
Programming Language
- Python
Speech Recognition
- Google Speech API
- CMU Sphinx (Offline)
Audio Processing
- PyAudio
NLP
- Text preprocessing
- Phrase normalization
- Token matching
Frontend
- EasyGUI
Clone the repository:
git clone https://github.com/aanxieee/isl_connect
Navigate to the project folder:
cd isl_connect
Install dependencies:
pip install -r requirements.txt
Run the application:
python main.py
- Launch the application
- Click the Record button
- Speak any phrase
- The system converts speech → text → ISL gestures
- Say "goodbye" to exit
Speech Input
"Hello how are you"
↓
Speech Recognition
"hello how are you"
↓
Phrase Detection
Matches ISL dictionary phrase
↓
Output
ISL GIF sequence displayed on screen
This project demonstrates how AI can be used to improve accessibility:
• Enables communication with deaf individuals using ISL
• Reduces dependency on interpreters
• Provides a scalable architecture that can expand to full vocabulary ISL translation
Future versions could include:
- Deep learning sign generation
- Real-time webcam gesture recognition
- Mobile deployment
- Transformer-based speech models