On-device assistive vision for the visually impaired — real-time object detection, face recognition, voice assistant, distance estimation, collision warnings, and spoken navigation alerts, running entirely on your Android phone.
Note
Looking for the Python server & SDK?
The server-based Python architecture (FastAPI, uvicorn, BoT-SORT, Ultralytics) has been moved to the legacy branch:
👉 View ai-integration branch
Camera Frame → YOLOv8s TFLite + ML Kit Face Detection → Distance Estimator → Centroid Tracker → Risk Scorer → Overlay + TTS
- Detect — YOLOv8s runs on-device via TensorFlow Lite (CPU/NNAPI) for objects, ML Kit for faces
- Recognize — Detected faces are matched against saved embeddings using cosine similarity
- Estimate Distance — Bounding box height → approximate distance in metres
- Track — Centroid-based tracker assigns persistent IDs and estimates velocity
- Score Risk — Objects scored by distance, approach speed, and direction
- Alert — Risk-colored bounding boxes on screen + spoken TTS warnings
- Listen — Voice assistant processes spoken commands for saving and recalling memories
- Blind-Safe Navigation: Color-coded bounding boxes (🟢 safe → 🟡 info → 🟠 warning → 🔴 critical) with spoken alerts like "Person approaching fast ahead, 2.3 metres! Move right."
- 🆕 Voice Assistant: Push-to-talk microphone button. Say commands like "what do you see", "save face as John", "save location", "help".
- 🆕 Face Recognition: On-device face detection + re-identification using ML Kit. Save faces and recognize them later — all offline.
- 🆕 Memory System: Save faces, objects, contacts, locations, timestamps, and notes using voice commands. Browse everything in the Memory screen.
- Object Tracking: Persistent IDs across frames with velocity estimation — knows when objects are approaching vs stationary.
- On-Device Inference: No server, no Wi-Fi needed. Runs entirely on your phone's CPU/NNAPI.
- Dual Camera: Use the phone camera (CameraX) or a wearable ESP32-CAM via Wi-Fi WebSocket/MJPEG stream.
- Adaptive Alerts: Scene modes (INDOOR/OUTDOOR), configurable sensitivity, cooldown timers, and TTS speech rate.
| Command | What It Does |
|---|---|
| "save face" / "save face as John" | Saves the detected face with a name |
| "who is this?" | Identifies the face against saved faces |
| "save this" | Saves the top detected object |
| "save contact [name]" | Saves a contact |
| "save location" | Saves current GPS coordinates |
| "save time" | Saves a timestamp with optional note |
| "save note [text]" | Saves a free-form note |
| "what do you see?" | Describes all detected objects aloud |
| "list memories" | Reads a summary of all saved items |
| "what time is it?" | Reads the current date and time |
| "battery level" | Reads device battery percentage |
| "navigate to [place]" | Gemini-powered safety-aware walking directions |
| "find [object]" / "where is [object]" | Automatically retrieves spatial context from memory |
| "repeat that" | Replays the last spoken response |
| "stop" / "be quiet" | Silences TTS immediately |
| "ask Gemini [question]" | Free-form Q&A with Gemini AI |
| any other question | Automatically routed to Gemini with memory & scene context |
| "help" | Lists available commands |
- Get a free API key from aistudio.google.com
- Open GlassInterface → Settings → paste key in Gemini API Key field
- Done! The assistant dynamically searches your saved Memory logs if you ask it to locate an item you previously encountered.
- Download
GlassInterface-V0.6.0.apkfrom Releases. - Transfer to your Android device and install (enable "Install from unknown sources").
git clone https://github.com/AnuranjanJain/GlassInterface.git
cd GlassInterface
./gradlew assembleDebug
# APK → app/build/outputs/apk/debug/app-debug.apkGlassInterface can act as a wearable headset processor using an ESP32-CAM:
- Flash ESP32-CAM with a standard
CameraWebServersketch. - Connect phone to the ESP32's Wi-Fi AP (or same local network).
- Open GlassInterface → Settings → Enable Use External Wi-Fi Camera.
- Enter the stream URL (default:
ws://192.168.4.1/Camera).
app/ # MainActivity, MainViewModel, Jetpack Compose UI
core/
├── ai-bridge/ # LocalAIEngine, FaceRecognitionEngine, CentroidTracker, RiskScorer
├── camera/ # CameraX FrameProvider, MjpegInputStream
├── common/ # AlertConfig, BoundingBox, DetectionResult data models
├── memory/ # Room database, entities, DAO, MemoryRepository
├── overlay/ # BoundingBoxOverlay (risk-colored Canvas drawing)
├── tts/ # TTSManager (cooldown-aware Text-to-Speech)
└── voice/ # VoiceInputManager, VoiceCommandParser, VoiceCommand
feature/
├── memory/ # Memory browser UI (faces, objects, contacts, locations, notes)
└── settings/ # Settings UI, DataStore repository
| Component | Technology |
|---|---|
| AI Model | YOLOv8s → TensorFlow Lite |
| Face Detection | Google ML Kit Face Detection |
| Language | Kotlin |
| UI | Jetpack Compose + Material 3 |
| DI | Dagger Hilt |
| Database | Room (SQLite) |
| Camera | CameraX + WebSocket/MJPEG streaming |
| Voice | Android SpeechRecognizer |
| Audio | Android TTS |
| Location | Play Services Location |
| Build | Gradle (Kotlin DSL) |