An offline multilingual audio intelligence system designed to analyze noisy real-world speech recordings containing Kashmiri, Hindi, and Urdu — often mixed within the same conversation — and convert them into structured, actionable intelligence.
The system functions as an autonomous triage agent that processes, analyzes, and prioritizes recordings without human intervention by generating structured intelligence reports and risk classifications.
- Processes noisy field audio
- Handles Hindi and Urdu robustly using multilingual modeling
- Handles Kashmiri using multilingual approximation (low-resource language support)
- Uses Faster-Whisper (Whisper large-v3, int8 CPU mode) for robust multilingual transcription
- Detects primary spoken language within each recording
- Generates timestamped transcript segments
Note: Kashmiri is a low-resource language. Performance depends on multilingual generalization rather than language-specific fine-tuning.
- Converts transcripts into clear English
- Uses Whisper translation mode and an additional MarianMT multilingual-to-English model for normalization
The system performs rule-based NLP extraction of structured intelligence:
| Category | Extracted Information |
|---|---|
| People | Names mentioned in conversation |
| Locations | Cities, regions, landmarks |
| Time References | Dates, times, schedules |
| Threat Keywords | Words such as attack, blast, weapon |
| Flagged Individuals | Names that co-occur with threat-related language |
Threat detection is based on customizable keyword matching rather than semantic embedding or contextual sentiment modeling.
Risk classification is determined using rule-based scoring combining threat keyword presence and sensitive location matching.
| Risk Level | Criteria |
|---|---|
| HIGH | Threat keywords present + sensitive location match |
| MEDIUM | Threat keywords present |
| LOW | No threat indicators |
The scoring system is deterministic and threshold-based. It does not use machine learning classification.
Audio Input
↓
Preprocessing (Noise handling + WAV conversion)
↓
Speech-to-Text (Faster-Whisper)
↓
Primary Language Detection
↓
Translation to English
↓
NLP Analysis (NER + Keyword Matching)
↓
Individual Flagging Logic
↓
Rule-Based Risk Scoring
↓
Structured Intelligence Report
Kashmir_Audio_Intelligence_Agent/
│
├── app.py # Streamlit UI entry point
├── requirements.txt # Project dependencies
├── README.md
│
├── audio_samples/ # Example/test audio files
│
├── data/
│ ├── threat_keywords.txt # Static threat keyword list
│ └── sensitive_locations.txt # Sensitive region list
│
├── hf_cache/ # HuggingFace model cache (created after first run)
│
├── src/
│ ├── preprocess.py # Audio conversion & cleanup
│ ├── transcribe.py # Whisper transcription engine
│ ├── translate.py # MarianMT translation module
│ ├── keywords.py # Keyword-based threat detection
│ ├── ner.py # Named Entity Recognition (spaCy)
│ ├── risk_agent.py # Rule-based risk scoring & flagging logic
│ └── report.py # Transcript highlighting
│
├── temp/ # Temporary audio processing files
└── venv/ # Virtual environment (recommended)
Audio files are converted to:
- Mono channel
- 16kHz sample rate
This standardization improves transcription consistency.
Faster-Whisper (large-v3, int8 CPU mode) performs:
- Multilingual transcription
- Primary language detection
- Voice Activity Detection (VAD)
- Timestamped segment generation
Transcripts are translated into English using:
- Whisper translation mode
- MarianMT multilingual-to-English model for normalization
Transcript text is scanned against a static, customizable keyword list.
Detection is literal string matching.
Using spaCy multilingual model:
- PER → People
- LOC/GPE → Locations
- DATE/TIME → Time references
Individuals are flagged for contextual review when their names co-occur with threat-related language.
Co-occurrence does not imply guilt. Flagging is purely heuristic and designed for analyst prioritization.
A deterministic scoring mechanism combines:
- Presence of threat keywords
- Matching against predefined sensitive locations
Final output is LOW, MEDIUM, or HIGH risk.
Built with Streamlit for offline desktop usage:
- Upload multiple audio files
- View English transcript
- Highlight threat keywords
- Display structured JSON intelligence report
- Batch risk summary across files
The system operates autonomously once files are uploaded.
git clone <repo-link>
cd kashmir-audio-agenta. On Windows:
python -m venv venv
venv\Scripts\activateb. On macOS / Linux:
python3 -m venv venv
source venv/bin/activatepython -m pip install --upgrade pippip install -r requirements.txtpython -m spacy download xx_ent_wiki_smInternet is required only for the initial download of:
- Faster-Whisper large-v3
- MarianMT multilingual-to-English model
These models are cached locally inside hf_cache/.
All subsequent executions operate fully offline.
Ensure your virtual environment is activated, then run:
streamlit run app.pyStreamlit will provide a local URL (typically http://localhost:8501) in the terminal.
- Requires internet only for initial model download
- Fully offline after caching
- CPU-optimized (int8 mode)
- Designed for low-connectivity environments
The system produces structured intelligence reports and batch risk summaries for analyst review.
- Kashmiri is a low-resource language; transcription quality may vary.
- Mixed-language code-switching is not explicitly segmented.
- Threat detection is keyword-based and not semantic.
- Risk scoring is rule-based and not ML-classified.
- No speaker identification or verification.
- Performance may degrade under heavy background noise.
- Audio triage for multilingual environments
- Preliminary intelligence prioritization
- Structured analysis of mixed-language recordings
- Offline threat keyword monitoring
This tool provides automated prioritization and risk estimation for analyst review.
It does not determine guilt, intent, or legal responsibility.
By Jairaj R, Ham P R, Ajo Jose