Skip to content

Diwakar-odds/Ai_Assistant

BranchesTags

Repository files navigation

🤖 YourDaddy AI Assistant

Version Python License Platform Status

"Command the chaos. Speak. Done. Next."

A sophisticated AI-powered personal assistant featuring voice recognition, smart automation, multilingual support, and real-time AI responses powered by Google Gemini and OpenAI.

Features • Quick Start • Installation • Configuration • Documentation

Last Updated: December 21, 2025

📋 Table of Contents

🌟 Overview

YourDaddy AI Assistant is a comprehensive, voice-activated AI assistant that combines the power of Google Gemini 2.0, OpenAI GPT, advanced speech recognition, and intelligent automation to create a seamless personal assistant experience. Built with Python and modern web technologies, it offers multilingual support, multimodal capabilities (text, voice, vision), and extensive system integration. It features a sophisticated dual-backend architecture to handle both real-time user interaction and complex, offline machine learning tasks.

🎯 What Makes This Special

  • 🧠 Real-Time AI: Intelligent responses using Google Gemini 2.0 Pro and OpenAI GPT-3.5/4
  • 🎤 Voice Control: Google Assistant-quality voice recognition with wake word detection
  • 🌍 Multilingual: Support for English, Hindi, Hinglish, and multiple Indian languages
  • 👁️ Vision AI: Image analysis, OCR, and visual understanding
  • 🔄 Smart Automation: Windows app control, file operations, and task scheduling
  • 🎵 Media Integration: Spotify and YouTube Music control with voice commands
  • 🌐 Modern Web UI: React + TypeScript interface with real-time WebSocket updates
  • 🔒 Secure: PIN-based authentication and encrypted API key management
  • 📚 Learning System: Adaptive learning with feedback loop and pattern recognition
  • 🚫 Offline Capable: Core features work without internet using Vosk and local models

✨ Key Features

🤖 AI & Intelligence

  • Conversational AI: Context-aware conversations with memory persistence
  • Google Gemini 2.0: Latest multimodal AI models with vision support
  • OpenAI GPT Integration: GPT-3.5 Turbo and GPT-4 support
  • Advanced Memory System: Long-term context retention and relationship mapping
  • Adaptive Learning: Pattern recognition and personalized responses with feedback loop
  • Knowledge Graphs: Intelligent associations and context understanding
  • Feedback Learning: User feedback integration for continuous improvement
  • Graph Neural Networks: Advanced learning with relationship mapping
  • Intent Classification: Smart command understanding with neural networks
  • Learning Dashboard: Real-time visualization of learning progress and metrics

🎙️ Voice & Speech

  • Wake Word Detection: "Hey Daddy", "OK Daddy" activation with local processing
  • Advanced Speech Recognition:
    • OpenAI Whisper API (online, highest accuracy)
    • Google Cloud Speech-to-Text
    • Vosk offline recognition (English & Hindi)
    • Browser-based Web Speech API
  • Neural Text-to-Speech:
    • Microsoft Edge-TTS (natural neural voices)
    • Google Cloud TTS
    • Coqui TTS (offline fallback)
    • pyttsx3 (system voices)
  • Multilingual Support: English, Hindi, Hinglish, Spanish, French, and more
  • Continuous Listening Mode: Always-on voice detection with smart activation
  • Voice Activity Detection: Advanced VAD with spectral feature analysis
  • Speaker Verification: Voice profile management for secure access

🖥️ System Integration & Automation

  • Smart App Discovery: Automatic detection of 500+ installed Windows applications
  • Application Control: Launch, close, and manage any Windows application
  • File Operations: Create, move, copy, search, organize files and folders
  • Taskbar Detection: Real-time window and application monitoring
  • System Control: Volume, brightness, power management
  • Automation Tools: Scheduled tasks, batch operations, system optimization

📅 Productivity & Communication

  • Google Calendar Integration: Event creation, reminders, scheduling
  • Email Automation: Send, read, and manage emails via voice or text
  • Task Scheduling: APScheduler-based automation with cron-like syntax
  • Document OCR: Extract text from images and PDFs using Tesseract
  • Web Scraping: Intelligent data extraction from websites
  • News & Weather: Real-time news updates and weather forecasts
  • Stock & Crypto Prices: Financial data monitoring

🎵 Media & Entertainment

  • Spotify Integration:
    • Play, pause, skip, control playback
    • Create and manage playlists
    • Search songs, artists, albums
    • Get music recommendations
  • YouTube Music: Search and play music videos
  • Music Downloader: yt-dlp integration for audio/video downloads
  • Media Player Control: System-wide media control

🌐 Modern Web Interface

  • React + TypeScript Frontend: Modern, responsive UI built with Vite
  • Real-Time Communication: WebSocket support for live updates
  • Flask Backend: RESTful API with Flask-SocketIO
  • Mobile-Friendly: Responsive design works on all devices
  • Dark/Light Themes: Customizable appearance
  • Voice Web Commands: Browser-based voice interaction
  • Live Status Updates: Real-time system and AI status monitoring
  • Application Grid: Smart app discovery and control interface
  • Conversation Space: Modern chat interface with enhanced UX
  • Learning Dashboard: Interactive visualization of learning metrics and feedback

👁️ Multimodal & Vision

  • Image Analysis: Visual understanding using Gemini Vision API
  • Video Processing: Frame extraction and multi-frame analysis
  • Screen Capture Analysis: Screenshot understanding and OCR
  • Object Detection: Identify objects, faces, and scenes
  • Document Understanding: Analyze document structure and content
  • Batch Processing: Process multiple images/documents simultaneously

🔒 Security & Privacy

  • PIN-Based Authentication: Secure access control with configurable PIN
  • Encrypted Credentials: Secure storage of API keys and tokens
  • Environment Variables: Secure configuration management
  • Rate Limiting: API request throttling and protection
  • JWT Tokens: Secure session management
  • Local Processing: Offline modes protect privacy

📱 Mobile & Cross-Platform Support NEW!

  • Progressive Web App (PWA): Install on iPhone/Android like a native app
  • Mobile-Optimized Interface: Touch-friendly, responsive design
  • Offline Mode: Service worker caching for offline access
  • Voice on Mobile: Browser-based voice commands on smartphones
  • Push Notifications: Get AI alerts on your phone
  • Cloud Deployment Ready: Deploy to Railway, Render, Heroku, etc.
  • Ngrok Integration: Instant internet access for testing
  • QR Code Setup: Scan and connect in seconds
  • Camera Integration: Send photos from phone to AI
  • Cross-Device Sync: Access from computer, phone, tablet

📲 Quick Mobile Access:

python setup_mobile.py  # One-click mobile setup

See MOBILE_PLATFORM_SUPPORT.md for complete mobile guide.

🚀 Quick Start (2 Minutes)

Step 1: Install Dependencies

# Clone the repository
git clone <repository-url>
cd assitant

# Install Python dependencies
pip install -r requirements.txt

Step 2: Setup API Keys (Free - 1 Minute)

# Run the quick setup wizard
python quick_ai_setup.py

# Get FREE Gemini API key:
# 1. Visit: https://aistudio.google.com/app/apikey
# 2. Sign in with Google account
# 3. Click "Create API Key"
# 4. Copy and paste when prompted

Step 3: Start the Assistant

# Start with web interface (recommended)
python main.py

# Start the modern web backend
python start_backend.bat  # Windows
# or
python ai_assistant/services/modern_web_backend.py

# Start React frontend (in new terminal)
cd src/project
npm install
npm run dev

# Or use specific interface
python main.py --interface web --port 8000
python main.py --interface cli

Step 4: Access & Test

🌐 Web Interface: http://localhost:8000
🚀 React Frontend: http://localhost:5173
🎤 Test Voice: "Hey Daddy, what's the weather today?"
💬 Test Chat: "Explain quantum computing"
📊 Learning Dashboard: http://localhost:8000/learning_dashboard.html

Expected: Intelligent AI responses (not templates!)

📱 Mobile & Cross-Platform Access

🚀 Use Your AI on Your Phone - Same Beautiful Interface!

Your React frontend now works as a Progressive Web App (PWA) - install it on your phone like a native app!

🎯 React Frontend Mobile Setup (RECOMMENDED):

Windows Quick Start:

scripts/launchers/start_react_mobile.bat  # One-click start!

Manual Start:

cd src/project
npm install  # First time only
npm run dev  # Starts on network: http://YOUR_IP:5173

On Your Phone:

  1. Open browser (Safari on iPhone, Chrome on Android)
  2. Go to http://YOUR_IP:5173 (shown in terminal)
  3. Wait 30 seconds - install prompt appears automatically
  4. Tap "Install" → App added to home screen!

Features:

  • ✅ Same beautiful React interface - not a separate mobile UI
  • ✅ Install on home screen - works like native app
  • ✅ Offline support - caches resources for offline use
  • ✅ Real-time dashboard - all features work on mobile
  • ✅ Network status indicator - see when offline
  • ✅ iOS + Android - works on all mobile browsers

📖 Detailed Guide: src/project/MOBILE_README.md 📋 Technical Summary: REACT_MOBILE_SUMMARY.md

Alternative: Standalone Mobile Server (Optional)

For a simpler, lightweight mobile interface:

# One-command mobile setup
python setup_mobile.py

Three Access Methods:

1️⃣ Local Network Access

  • Access via browser: http://YOUR_IP:5000
  • Same WiFi network required
  • Fast and private

2️⃣ Internet Access (Anywhere)

# Using Ngrok (instant internet access)
python mobile_server.py --ngrok

# Or deploy to cloud (Railway, Render, Heroku)
# See DEPLOYMENT_GUIDE.md

Mobile Features:

  • ✅ Voice commands on phone
  • ✅ Text chat interface
  • ✅ Camera integration for AI vision
  • ✅ File uploads from phone
  • ✅ Offline mode with caching

📚 Complete Mobile Documentation:

Need help? Run python setup_mobile.py for guided setup wizard!

🏗️ Architecture

This project utilizes a sophisticated dual-backend architecture to separate user-facing operations from developer-focused analytics and ML model interaction.

  • User-Facing Application: The primary application that users interact with. It consists of a modern React frontend and a Flask backend that handles real-time communication, automation, and core assistant logic.
  • Learning & Analytics API: A separate FastAPI backend that exposes the 27+ advanced machine learning systems via a comprehensive REST API. This server is intended for developers and data scientists to monitor, debug, and analyze the AI's learning processes.

Crucially, these two backends operate independently. The main Flask application does not call the FastAPI server. Instead, both backends act as two different "heads" for the same shared core of ML modules.

                  ┌──────────────────┐      ┌────────────────────┐
                  │      User        │      │     Developer /    │
                  └──────────────────┘      │      Analyst       │
                         │                  └────────────────────┘
     /-------------------|---------------------------\
     |                   |                           |
┌────▼──────────┐ ┌──────▼──────────────┐ ┌──────────▼───────────┐
│ React + Vite  │ │ Flask + SocketIO    │ │  FastAPI (Uvicorn)   │
│  (Frontend)   │ │ (User-Facing Backend) │ │ (Learning & Analytics │
│ (src/project/)│ │(modern_web_backend.py)│ │         API)         │
└──────┬────────┘ └──────────┬──────────┘ │ (learning_api.py)    │
       │                      │            └──────────┬───────────┘
       └───────────┬──────────┘                       │
                   |                                  |
           ┌───────▼──────────────────────────────────▼────────┐
           │     Core AI / ML Systems (Shared Python Library)    │
           │        (27+ Learning Modules in src/ai_assistant/ai) │
           └──────────────────────────┬──────────────────────────┘
                                      │
                 ┌────────────────────┴────────────────────┐
                 │          Data Persistence Layer         │
                 │(7+ SQLite Databases in /data directory) │
                 └────────────────────┬────────────────────┘
                                      │
  ┌──────────┬──────────┬─────────────┴──────────┬──────────┬──────────┐
  │          │          │                        │          │          │
┌─▼──┐   ┌───▼──┐   ┌───▼──┐              ┌──────▼──┐  ┌─────▼──┐  ┌────▼───┐
│Gemini│ │OpenAI│ │Spotify │              │ Windows │  │ Google │  │ Voice  │
│ API  │ │ API  │ │  API   │              │   OS    │  │Services│  │Engines │
└──────┘ └──────┘ └────────┘              └─────────┘  └────────┘  └────────┘

📦 Detailed Installation

Prerequisites

  • Python: 3.8 or higher
  • OS: Windows 10/11 (primary), Linux/Mac (experimental)
  • RAM: 4GB minimum, 8GB recommended
  • Storage: 2GB for dependencies and models

Complete Installation Steps

1. Clone Repository

git clone <repository-url>
cd assitant

2. Create Virtual Environment (Recommended)

# Windows
python -m venv venv
venv\Scripts\activate

# Linux/Mac
python3 -m venv venv
source venv/bin/activate

3. Install Dependencies

# Install all requirements
pip install -r requirements.txt

# For development (includes testing tools)
pip install -r requirements.txt pytest pytest-cov black flake8

4. Install Vosk Models (For Offline Voice)

# Download English model
python -c "from ai_assistant.voice import download_vosk_model; download_vosk_model('en')"

# Download Hindi model
python -c "from ai_assistant.voice import download_vosk_model; download_vosk_model('hi')"

5. Install System Dependencies (Windows)

# Install Tesseract OCR (for document processing)
# Download from: https://github.com/UB-Mannheim/tesseract/wiki
# Add to PATH: C:\Program Files\Tesseract-OCR

# PyAudio (for microphone access)
pip install pipwin
pipwin install pyaudio

6. Setup Configuration

# Run setup wizard
python quick_ai_setup.py

# Or manually create config files
cp config/app_integration.env.example config/app_integration.env
cp config/multimodal_config.json.example config/multimodal_config.json

7. Setup API Keys

Option A: Using Setup Wizard (Recommended)

python quick_ai_setup.py

Option B: Manual Setup Create api_keys.json:

{
    "GEMINI_API_KEY": "your-gemini-api-key",
    "OPENAI_API_KEY": "your-openai-api-key",
    "SPOTIFY_CLIENT_ID": "your-spotify-client-id",
    "SPOTIFY_CLIENT_SECRET": "your-spotify-client-secret"
}

8. Verify Installation

# Check dependencies
python check_dependencies.py

# Test AI integration
python test_ai_quick.py

# Test voice recognition
python test_import_core.py

🔧 Configuration

API Keys Configuration

Required Keys

  1. Gemini API (Free - Recommended)

  2. OpenAI API (Optional - Paid)

Optional Keys

  1. Spotify API (Free)

  2. Google Cloud (Free tier available)

Configuration Files

1. api_keys.json (Recommended)

{
    "GEMINI_API_KEY": "AIza...",
    "OPENAI_API_KEY": "sk-...",
    "SPOTIFY_CLIENT_ID": "your-client-id",
    "SPOTIFY_CLIENT_SECRET": "your-client-secret",
    "GOOGLE_CLOUD_KEY_PATH": "path/to/credentials.json"
}

2. config/user_settings.json

{
    "language": "en",
    "voice_enabled": true,
    "wake_word": "hey daddy",
    "tts_engine": "edge-tts",
    "theme": "dark"
}

3. config/multimodal_config.json

{
    "vision_enabled": true,
    "max_image_size": 4096,
    "ocr_enabled": true,
    "video_processing": false
}

4. Environment Variables (Alternative)

# Windows
set GEMINI_API_KEY=your-key-here
set OPENAI_API_KEY=your-key-here

# Linux/Mac
export GEMINI_API_KEY=your-key-here
export OPENAI_API_KEY=your-key-here

Security Settings

Setup PIN Authentication

# First time setup
python setup_pin.py

# Or during startup
python main.py --setup-pin

# Skip for development (not recommended)
python main.py --skip-auth

🎮 Usage

Starting the Assistant

Web Interface (Recommended)

# Default (port 8000)
python main.py

# Custom port
python main.py --port 5000

# With verbose logging
python main.py --verbose

# Access at: http://localhost:8000

Command Line Interface

python main.py --interface cli

Desktop GUI

python main.py --interface desktop

Using Voice Commands

Wake Word Activation

Say: "Hey Daddy" or "OK Daddy"
Wait for: Activation sound/beep
Then say: Your command

Example Voice Commands

"Hey Daddy, what's the weather today?"
"Open Chrome and search for Python tutorials"
"Play some relaxing music on Spotify"
"Create a meeting for tomorrow at 3 PM"
"Send an email to john@example.com"
"What's in this image?" (with image upload)
"Translate 'Hello' to Hindi"
"Set a reminder for 5 PM"

Using Text Commands

Web Interface

  1. Open http://localhost:8000
  2. Type your query in the chat box
  3. Press Enter or click Send

CLI Interface

$ python main.py --interface cli
> What's the capital of France?
> Open notepad
> Play music by Coldplay
> quit  # To exit

Advanced Usage

Automation Scripts

# Schedule automated tasks
python scripts/setup/setup_automation.py

# Run batch file operations
python -c "from ai_assistant.file_ops import organize_files_by_type; organize_files_by_type('~/Downloads')"

API Endpoints

Health Check

curl http://localhost:8000/api/health

Send Message

curl -X POST http://localhost:8000/api/chat \
  -H "Content-Type: application/json" \
  -d '{"message": "Hello!", "user_id": "user123"}'

Get Features

curl http://localhost:8000/api/features

Voice Recognition

curl -X POST http://localhost:8000/api/voice/recognize \
  -F "audio=@recording.wav"

📁 Project Structure

assitant/
├── main.py                          # Main entry point with interface selection
├── check_deps.py                    # Dependency checker
├── modern_web_backend.py            # Local backend entry
├── test_voice_enhancement.py        # Voice testing script
├── fix_syntax.py                    # Syntax fixing utility
│
├── src/                             # Main Source Code Directory
│   ├── ai_assistant/                # Core AI logic package
│   │   ├── ai/                      # Real-time LLM operations
│   │   ├── automation/              # Windows and software automation
│   │   ├── core/                    # Core configs
│   │   ├── integrations/            # 3rd party APIs (Spotify, calendar, etc.)
│   │   ├── modules/                 # System modules
│   │   ├── voice/                   # Voice recognition and synthesis
│   │   ├── vision/                  # Vision API utilities
│   │   └── workflow/                # Multi-step tasks
│   │
│   ├── project/                     # Frontend React UI
│   │   ├── src/                     # React components
│   │   ├── public/                  # Static web assets
│   │   ├── package.json             # NPM dependencies
│   │   └── vite.config.ts           # Vite build config
│   │
│   ├── config/                      # Config parsers and defaults
│   ├── data/                        # Core data interfaces
│   ├── launchers/                   # Python launcher scripts
│   ├── user_data/                   # User specific resources
│   ├── utils/                       # Common utility functions
│   └── web_assets/                  # Assorted web files
│
├── scripts/                         # Utility & Setup Scripts
│   ├── setup/                       # Installation wizards and keystores
│   ├── launchers/                   # Bash and Batch starting scripts
│   ├── analysis/                    # Python scripts for analytics
│   ├── debug/                       # Logging operations
│   ├── learning/                    # Memory / Learning ops
│   ├── diagnostics/                 # Diagnostics tools
│   └── validation/                  # Pre-flight checks
│
├── tests/                           # Component and Integration Test Suite
├── config/                          # Workspace configuration files
├── data/                            # Core Runtime Datastores
├── databases/                       # SQLite backend files
├── model/                           # Local offline ML models
├── docs/                            # Extensive Documentations
├── logs/                            # Application Event Logs
└── deploy/                          # Deployment configurations

🔌 API Integrations

1. Google Gemini AI

  • Purpose: Primary AI conversational engine
  • Features: Text generation, vision analysis, multimodal understanding
  • Setup: Get free API key from https://aistudio.google.com/app/apikey
  • Usage: 60 requests/minute free tier

2. OpenAI

  • Purpose: Alternative LLM, Whisper speech recognition
  • Models: GPT-3.5-Turbo, GPT-4, Whisper API
  • Setup: https://platform.openai.com/api-keys
  • Cost: Pay-per-use (~$0.002/conversation)

3. Spotify

4. Google Cloud Services

  • Speech-to-Text: High-accuracy voice recognition
  • Text-to-Speech: Natural voice synthesis
  • Calendar API: Event management
  • Gmail API: Email automation
  • Setup: https://console.cloud.google.com

5. Vosk (Offline)

  • Purpose: Offline speech recognition
  • Languages: English, Hindi, 20+ others
  • Setup: Automatic model download
  • No API Key Required

🧪 Testing

Run All Tests

# Run all tests
pytest

# With coverage report
pytest --cov=ai_assistant --cov-report=html

# Specific test file
pytest tests/test_ai_quick.py

# Verbose output
pytest -v

Test Specific Components

AI Integration

python test_ai_quick.py
python test_real_ai.py

Voice Recognition

python test_import_core.py
python ai_assistant/voice/test_voice_recognition.py

All 27 Systems

python test_all_27_systems.py

Web Backend

python test_api_endpoints.py

Manual Testing

Test AI Response

from ai_assistant.ai import get_ai_response

response = get_ai_response("What is quantum computing?")
print(response)

Test Voice Recognition

from ai_assistant.voice import recognize_speech

text, confidence = recognize_speech()
print(f"You said: {text} (confidence: {confidence})")

Test Automation

from ai_assistant.automation import open_application

open_application("Chrome")

🔧 Troubleshooting

Common Issues

1. Import Errors

# Problem: ModuleNotFoundError
# Solution: Ensure virtual environment is activated and dependencies installed
pip install -r requirements.txt

2. API Key Not Working

# Check AI status
python check_ai_status.py

# Verify keys are loaded
python -c "from ai_assistant.ai import check_api_keys; check_api_keys()"

3. Voice Recognition Not Working

# Check microphone permissions (Windows)
# Settings > Privacy > Microphone > Allow apps

# Test PyAudio installation
python -c "import pyaudio; print('PyAudio OK')"

# Download Vosk models
python -c "from ai_assistant.voice import download_vosk_model; download_vosk_model('en')"

4. Web Interface Not Loading

# Check port availability
netstat -ano | findstr :8000

# Try different port
python main.py --port 5000

# Check firewall settings

5. Tesseract OCR Not Found

# Windows: Download and install from
# https://github.com/UB-Mannheim/tesseract/wiki

# Add to PATH or set environment variable
set TESSERACT_CMD=C:\Program Files\Tesseract-OCR\tesseract.exe

Debug Mode

# Enable verbose logging
python main.py --verbose

# Check logs
cat logs/assistant.log

# Debug specific module
python debug_launcher.py

Getting Help

  1. Check docs/ directory for detailed documentation
  2. Review TROUBLESHOOTING.md
  3. Check logs/ for error messages
  4. Run diagnostic tools: python check_dependencies.py

🤝 Contributing

We welcome contributions! See CONTRIBUTING.md for guidelines.

Development Setup

# Fork and clone repository
git clone https://github.com/yourusername/assitant.git

# Create feature branch
git checkout -b feature/amazing-feature

# Install development dependencies
pip install -r requirements.txt pytest black flake8

# Make changes and test
pytest
black ai_assistant/
flake8 ai_assistant/

# Commit and push
git commit -m "Add amazing feature"
git push origin feature/amazing-feature

📄 License

This project is licensed under the MIT License - see LICENSE.txt for details.

🙏 Acknowledgments

  • Google Gemini: Advanced AI capabilities
  • OpenAI: GPT and Whisper models
  • Vosk: Offline speech recognition
  • Edge-TTS: Neural text-to-speech
  • Flask & React: Web framework and UI
  • All Contributors: Thank you for your contributions!

📞 Support

Made with ❤️ by the YourDaddy AI Team

⭐ Star us on GitHub if you find this helpful!

About

No description, website, or topics provided.

Resources

Readme

Contributing

Contributing

Security policy

Security policy
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages