An intelligent web application for real-time American Sign Language (ASL) alphabet recognition using Computer Vision and Machine Learning
Features • Demo • Installation • Usage • Technology Stack
SmartSign is a real-time ASL alphabet recognition system that bridges communication gaps by translating hand gestures into text. Built with Flask, MediaPipe, and Machine Learning, this project demonstrates practical applications of Computer Vision in accessibility technology.
- Real-time Recognition: Instant ASL alphabet detection through webcam
- High Accuracy: MLPClassifier trained on thousands of hand landmark samples
- Smooth Predictions: Temporal smoothing with majority voting eliminates flickering
- Modern UI: Clean, responsive interface with live confidence scores
- Word Builder: Automatically constructs words from recognized letters
- Text-to-Speech: 🔊 Speaks completed words automatically (offline TTS)
- Production Ready: Optimized for performance (<50ms per frame)
- 🎥 Live Webcam Streaming - Real-time video processing at 30 FPS
- 🖐️ Hand Landmark Detection - MediaPipe extracts 21 3D landmarks per hand
- 🧠 ML Classification - Neural Network (MLPClassifier) predicts ASL letters
- 📊 Confidence Display - Visual confidence bar for predictions
- 📝 Word Builder - Auto-assembles letters into words with 1-second stability
- 🔊 Text-to-Speech - Speaks completed words using offline TTS (pyttsx3)
- 🎨 Animated Landing Page - Modern UI with gradient animations
- ⚡ Singleton Pattern - Efficient model loading (no redundant reinitializations)
- 🔄 Temporal Smoothing - 10-frame buffer for stable predictions
- 🎯 Confidence Filtering - Only displays predictions above 75% confidence
- 🗣️ Non-blocking Speech - Background thread TTS without FPS impact
- 🚀 Performance Optimized - FPS limiting and threaded camera handling
- 🛡️ Error Handling - Graceful degradation on camera/model failures
Input: Hand gesture (ASL letter 'A')
↓
MediaPipe detects 21 landmarks
↓
Feature extraction (63 coordinates + 11 engineered features)
↓
MLP Classifier predicts: "A" (confidence: 0.92)
↓
Temporal smoothing confirms stable prediction
↓
Output: Letter displayed + added to word builder
Add your screenshots here after deployment
- Landing page with animated gradients
- Live recognition with confidence display
- Word builder in action
- Python 3.8 or higher
- Webcam (for live recognition)
- Windows/Linux/macOS
-
Clone the repository
git clone https://github.com/YOUR_USERNAME/SmartSign.git cd SmartSign -
Create virtual environment
python -m venv venv # Activate on Windows venv\Scripts\activate # Activate on macOS/Linux source venv/bin/activate
-
Install dependencies
pip install -r requirements.txt
-
Verify model files exist
# Check models directory ls models/ # Should contain: asl_model.pkl, scaler.pkl, label_encoder.pkl
-
Start Flask server
python -m backend.app
-
Open in browser
http://127.0.0.1:5000/ -
Navigate to live demo
- Click "Start Live Demo" button
- Allow camera permissions
- Show ASL hand gestures to camera
If you have your own ASL dataset:
-
Prepare dataset
# Place images in data/raw/asl_alphabet_train/ # Format: data/raw/asl_alphabet_train/A/*.jpg
-
Extract landmarks
python -m training.extract_landmarks
-
Train model
python -m training.train_model
- Flask - Web framework for API endpoints and routing
- MediaPipe Hands - Real-time hand landmark detection
- scikit-learn - MLPClassifier for gesture classification
- OpenCV - Video streaming and frame processing
- NumPy - Numerical computations
- HTML5/CSS3 - Modern UI with glassmorphism effects
- JavaScript - Real-time prediction polling and updates
- Fetch API - Asynchronous backend communication
- Feature Engineering: 74 features (63 coords + 4 distances + 2 velocity + 5 angles)
- Normalization: StandardScaler for feature scaling
- Model: MLPClassifier (128, 64 hidden layers)
- Smoothing: EMA + majority voting
SmartSign/
├── backend/
│ ├── app.py # Flask application & routes
│ ├── processor.py # Feature extraction & prediction
│ ├── inference.py # Singleton predictor interface
│ └── feature_pipeline.py # Feature engineering utilities
├── frontend/
│ └── templates/
│ ├── index.html # Live recognition page
│ └── landing.html # Landing page
├── models/
│ ├── asl_model.pkl # Trained MLPClassifier
│ ├── scaler.pkl # StandardScaler
│ └── label_encoder.pkl # Label encoder
├── training/
│ ├── extract_landmarks.py # Extract MediaPipe landmarks
│ └── train_model.py # Train ML model
├── data/
│ ├── raw/ # Dataset images
│ └── processed/ # Extracted landmarks CSV
├── requirements.txt # Python dependencies
└── README.md # Documentation
MediaPipe detects hand in real-time and extracts 21 3D landmarks:
- Wrist (1 landmark)
- Thumb (4 landmarks)
- Index, Middle, Ring, Pinky fingers (4 landmarks each)
# Normalized coordinates (63 features)
coords = flatten(landmarks - wrist) / hand_size
# Distance features (4 features)
distances = [thumb_to_index, index_to_middle, middle_to_ring, ring_to_pinky]
# Finger angles (5 features)
angles = calculate_bending_angles(landmarks)
# Total: 63 + 4 + 2 + 5 = 74 features1. Extract features from frame
2. Apply StandardScaler normalization
3. Pass to MLPClassifier
4. Get probability distribution
5. Filter by confidence threshold (0.75)
6. Apply temporal smoothing (10-frame buffer)
7. Return most frequent prediction- Detects stable letter (held for 1 second)
- Appends to word display
- User can copy or clear words
- Training Accuracy: ~95%
- Test Accuracy: ~92%
- Inference Speed: <50ms per frame
- Frame Rate: 30 FPS
- Classes: 29 (A-Z + del + space + nothing)
- Add support for Indian Sign Language (ISL)
- Implement dynamic gesture recognition (words/phrases)
- Mobile app with React Native
- Cloud deployment (AWS/Heroku)
- Multi-user session support
- Voice output for recognized words
- Save conversation history
Contributions are welcome! Here's how you can help:
- Fork the repository
- Create feature branch (
git checkout -b feature/AmazingFeature) - Commit changes (
git commit -m 'Add AmazingFeature') - Push to branch (
git push origin feature/AmazingFeature) - Open Pull Request
This project is licensed under the MIT License - see the LICENSE file for details.
- MediaPipe - Hand landmark detection
- ASL Alphabet Dataset - Training data
- Flask, scikit-learn, and OpenCV communities
If you found this project helpful, please ⭐ star the repository!
For issues or questions, please open an issue.
Made with ❤️ for accessibility and inclusion