NeuroCursor

Hands-Free Computer Control with Face Gestures and Voice

NeuroCursor is an accessibility-focused desktop application that enables hands-free computer control using facial gestures and voice feedback. Perfect for users with limited arm mobility or anyone who wants hands-free computer interaction.

Features

• Left Click: Voice command “click”
• Right Click: Voice command “right click”
• Double Click: Long blink (both eyes closed for 0.3-0.5 seconds)
• Cursor Movement: Head movement left/right (and optionally up/down)
• Scroll Up: Puff cheeks (cheek landmarks move outward)
• Scroll Down: Smile / big grin (mouth corners stretch wide)
• Voice Feedback: Text-to-speech announcements for all actions
• Voice Assistant: Clone your own voice via Fish Audio
• Voice Commands & Dictation: Say “click”, “right click”, or “dictation” to control the mouse and enter text by voice
• Calibration: Personalize head position and gesture sensitivity

Requirements

• Python 3.9 or higher
• Webcam
• Microphone (for voice commands/dictation)
• Windows, macOS, or Linux

Installation

1. Clone or download this repository

2. Install dependencies:

pip install -r requirements.txt

3. (Optional) Set up Fish Audio API for enhanced voice feedback:
   • Get your API key from Fish Audio
   • Set environment variable: export FISH_AUDIO_API_KEY=your_key_here
   • If not set, the app will use system TTS as fallback

Project Structure

• backend/ – core tracking engine (neurocursor.py), camera test script, shared logic
• frontend/ – tkinter GUI (neurocursor_gui.py)
• PROJECT_PLAN.md, README.md, QUICKSTART.md – documentation

Usage

Command Line Version

Run the main application:

python backend/neurocursor.py

Controls:

• c - Calibrate (look straight at screen)
• e - Enable/Disable gesture control
• q - Quit

GUI Version

Run the graphical interface:

python -m frontend.neurocursor_gui

Steps:

1. Click "Start NeuroCursor"
2. Click "Calibrate" and look straight at the screen
3. Click "Enable Gestures"
4. Use gestures (smiles, cheek puffs, blinks) for movement/scroll, and voice commands (say “click”, “right click”, “dictation”) for clicks and typing

Test Camera

Test your webcam first:

python backend/test_camera.py

Gesture Guide

Gesture	Action
Voice: “Click”	Left Click
Voice: “Right click”	Right Click
Long Blink (0.3-0.5s)	Double Click
Head Left/Right	Move Cursor
Smile / Big Grin	Scroll Down
Cheek Puff	Scroll Up

Voice Assistant

• In the GUI, open the Voice Assistant panel and choose between the built-in NeuroCursor Voice or your personal My Voice (after cloning).
• Click Preview Voice to hear “Hello, welcome to NeuroCursor” before committing.
• Click Clone My Voice… to upload a short audio sample (30–60 seconds). NeuroCursor will use Fish Audio to create a custom narrator and save it in voice_config.json.
• You can switch voices at any time; each action (“Left click”, “Warning: face lost”, etc.) will be spoken in the active voice.

Voice Commands & Dictation

• NeuroCursor listens in the background (requires a microphone and the SpeechRecognition package).
• Say “click” for a left click, “right click” for a right click.
• Say “dictation” / “dictate this” to enter dictation mode, speak your sentence, and it will be typed into the active field automatically. Say “stop dictation” to cancel.
• Voice control runs alongside gesture control, so you can mix speech and head movements as needed. (The default recognizer uses Google Web Speech, so an internet connection is required.)

Calibration

Calibration is important for accurate cursor control:

1. Sit in your normal position
2. Look straight at the screen
3. Press c (or click "Calibrate" in GUI)
4. Stay still for 1 second
5. Calibration data is saved and loaded automatically

Configuration

You can adjust thresholds in backend/neurocursor.py:

• BLINK_THRESHOLD: Eye closed threshold (default: 0.25)
• OPEN_THRESHOLD: Eye open threshold (default: 0.30)
• DEAD_ZONE: Cursor movement dead zone in pixels (default: 30)
• MAX_SPEED: Maximum cursor speed (default: 10 pixels/frame)
• CURSOR_SENSITIVITY: Multiplier for cursor movement (default: 3.0)

Troubleshooting

Camera not detected:

• Make sure your webcam is connected and not used by another application
• Try running backend/test_camera.py first

Gestures not working:

• Ensure good lighting
• Make sure you've calibrated (c key)
• Check that gestures are enabled (e key)
• Adjust thresholds if needed

Cursor movement too sensitive/not sensitive enough:

• Recalibrate in your normal position
• Adjust DEAD_ZONE and MAX_SPEED in the code

Development

The project is organized in phases:

• Phase 1: Setup & Camera
• Phase 2: Face Tracking
• Phase 3: Wink Detection & Clicks
• Phase 4: Head Movement → Cursor
• Phase 5: Scroll & Double Click
• Phase 6: Fish Audio TTS
• Phase 7: GUI & Calibration
• Phase 8: Polish

See PROJECT_PLAN.md for detailed specifications.

License

This project is open source and available for use.

Contributing

Contributions welcome! Please feel free to submit issues or pull requests.

Acknowledgments

• MediaPipe for face mesh detection
• Fish Audio for TTS capabilities
• Built for accessibility and inclusion