OS Anomaly Sentinel

Adaptive Anomaly Detection in Operating Systems Using Machine Learning

Replace brittle fixed CPU/memory thresholds with unsupervised ML that learns per-machine normal behaviour. Collect process-level telemetry, engineer statistical features, train Isolation Forest, and detect anomalies — all without labelled data, kernel modifications, or process termination.

1. Project Overview

OS Anomaly Sentinel monitors running processes (CPU, memory, threads, status) via psutil, aggregates them into statistical time windows, and uses unsupervised machine learning (Isolation Forest) to detect anomalous behaviour relative to each machine's learned baseline. A 10-page Streamlit dashboard provides live monitoring, training, detection, multi-model evaluation, investigation, and auto-generated final-year reports.

Key insight: Every machine has a different "normal." Fixed thresholds (CPU > 80 %) generate false positives on busy machines and miss anomalies on quiet ones. Adaptive baselines solve both problems.

2. Problem Statement

Traditional OS monitoring uses static rules like "alert if CPU > 80%". These fail because:

Every machine has a different normal baseline
Busy machines generate false positives
Quiet machines miss real anomalies
Gradual drifts (memory leaks) go undetected
Thresholds require manual tuning per deployment

3. Proposed Solution

OS Anomaly Sentinel learns each machine's normal behaviour using unsupervised ML (Isolation Forest). Anomalies are detected as statistical deviations from the learned baseline, not against arbitrary thresholds.

Important: The main anomaly detector (Isolation Forest) does not rely on hard-coded thresholds. Fixed CPU/memory thresholds are included only as a baseline for comparison during evaluation (see Section 14).

4. Key Features

Adaptive baselines — model learns per-machine normal behaviour
Unsupervised learning — no labelled data required for training
Process-level telemetry — CPU, memory, threads, status via psutil
Feature engineering — 11 statistical features over time windows
Multi-model comparison — IF vs LOF vs One-Class SVM vs Fixed Threshold
Human-readable explanations — deviation-based with confidence levels
Interactive dashboard — 10-page Streamlit frontend with Plotly charts
Synthetic mode — cross-platform demo without real processes
Evaluation framework — precision, recall, F1, confusion matrix
Final-year reports — auto-generated project documentation
Viva preparation — 45+ likely questions with answers
Read-only — never terminates processes or modifies system state
Manual visual UI verification — All 10 dashboard pages visually inspected and screenshotted

5. Final-Year Project Highlights

Research contribution: Demonstrates that unsupervised ML (Isolation Forest) outperforms fixed-threshold monitoring for OS anomaly detection
Engineering quality: Modular architecture, comprehensive testing, configuration-driven pipeline, error handling
Academic rigour: Evaluation framework with 4-model comparison, confusion matrices, precision/recall/F1 analysis
Professional frontend: 10-page Streamlit dashboard with Plotly charts, metric cards, spinners, and error handling
Documentation: Auto-generated final-year report, abstract, technical architecture, and viva preparation material
Reproducibility: Synthetic data generator with 6 controlled anomaly types and ground-truth labels
Safety: Read-only monitoring — never terminates processes or modifies system state

6. System Architecture

Data Flow

Monitoring Agent → Raw Telemetry → Feature Engineering → Baseline Training
    → Anomaly Detection → Evaluation & Comparison → Streamlit Dashboard
    → Reports & Viva Material

Module-Level Flow

┌─────────────────────────────────────────────────────────────┐
│                     Data Collection                          │
│  psutil live  ──▶  data/raw/process_metrics.csv             │
│  synthetic    ──▶  (with ground-truth labels)               │
└───────────────────────────────┬─────────────────────────────┘
                                ▼
┌─────────────────────────────────────────────────────────────┐
│                   Feature Engineering                        │
│  Aggregate 5-min windows  ──▶  data/processed/features.csv  │
│  11 features: avg/max/std CPU & memory, trends, count       │
└───────────────────────────────┬─────────────────────────────┘
                                ▼
┌─────────────────────────────────────────────────────────────┐
│                Model Training & Detection                    │
│  Isolation Forest (scikit-learn)  ──▶  models/*.joblib       │
│  Anomaly scoring + severity + explanations                   │
│  ──▶  data/anomalies/anomaly_events.csv                     │
└───────────────────────────────┬─────────────────────────────┘
                                ▼
┌─────────────────────────────────────────────────────────────┐
│                   Evaluation & Reporting                     │
│  Compare 4 models (IF, LOF, OCSVM, Fixed)                   │
│  ──▶  data/reports/evaluation_results.csv                   │
│  ──▶  data/reports/model_comparison.md                      │
│  ──▶  data/reports/FINAL_YEAR_REPORT.md                     │
└───────────────────────────────┬─────────────────────────────┘
                                ▼
┌─────────────────────────────────────────────────────────────┐
│                   Streamlit Dashboard (10 pages)             │
│  Overview │ Live Monitor │ Training │ Detection │ Invest.   │
│  Model Eval │ Data Exp. │ Reports │ Settings │ Help        │
└─────────────────────────────────────────────────────────────┘

7. Tech Stack

Component	Technology
Language	Python 3.11+
Process collection	psutil
Data processing	pandas, NumPy, SciPy
ML models	scikit-learn (Isolation Forest, LOF, One-Class SVM)
Dashboard	Streamlit
Charts	Plotly
Configuration	PyYAML
Serialization	joblib
Notebook tooling	nbformat, ipykernel
Packaging	PyInstaller

8. Repository Structure

os-anomaly-sentinel/
├── README.md                             # This file
├── config.yaml                           # Pipeline configuration
├── requirements.txt                      # Python dependencies
├── OS_Anomaly_Detection_Project.ipynb    # Jupyter notebook (19 sections)
├── run_agent.py                          # Data collection agent
├── train_baseline.py                     # Model training script
├── detect.py                             # Anomaly detection script
├── evaluate.py                           # Model evaluation script
├── generate_report.py                    # Report generation script
├── package_app.py                        # PyInstaller packaging guide
├── dashboard.py                          # Streamlit entry point
│
├── src/
│   ├── collector.py                      # psutil process monitoring
│   ├── storage.py                        # CSV read/write utilities
│   ├── features.py                       # Feature engineering (11 features)
│   ├── model.py                          # Isolation Forest wrapper
│   ├── detector.py                       # Detection pipeline
│   ├── explanations.py                   # Human-readable explanations
│   ├── synthetic.py                      # Labeled synthetic data generator
│   ├── baselines.py                      # Comparison models (LOF, OCSVM, Fixed)
│   ├── evaluation.py                     # Model comparison metrics
│   ├── reporting.py                      # Report generation
│   ├── utils.py                          # Logging, directory helpers
│   └── ui/                               # Streamlit frontend
│       ├── styles.py                     # CSS injection
│       ├── state.py                      # Session state, paths, file sync
│       ├── actions.py                    # Backend wrappers
│       ├── components.py                 # Reusable UI components
│       ├── charts.py                     # Plotly chart functions
│       └── layout.py                     # 10 page renderers
│
├── tests/
│   ├── validate_notebook.py              # Notebook structure validator
│   ├── test_features.py                  # Feature engineering tests
│   ├── test_model.py                     # Model training tests
│   ├── test_synthetic.py                 # Synthetic data tests
│   ├── test_detector.py                  # Detection pipeline tests
│   ├── test_evaluation.py                # Evaluation tests
│
├── data/
│   ├── raw/                              # Raw telemetry CSV
│   ├── processed/                        # Aggregated features
│   ├── anomalies/                        # Detection results
│   └── reports/                          # Generated reports
│       ├── FINAL_YEAR_REPORT.md          # Full project report
│       ├── PROJECT_ABSTRACT.md           # One-page abstract
│       ├── viva_questions.md             # 45+ viva Q&A pairs
│       ├── technical_architecture.md     # Architecture deep-dive
│       ├── demo_summary.md               # 10-step demo script
│       ├── model_comparison.md           # IF vs LOF vs OCSVM vs Fixed
│       ├── evaluation_results.csv        # Per-model metrics
│       ├── evaluation_summary.json       # JSON summary
│       ├── confusion_matrix.csv          # TN, FP, FN, TP per model
│       ├── MANUAL_UI_VERIFICATION_REPORT.md  # Full manual visual QA report (PASS)
│       └── screenshots/                  # 10 page screenshots
│           ├── 01_overview.png
│           ├── 02_live_monitoring.png
│           └── ... (through 10_help_about.png)
│
├── models/                               # Trained model artifacts
│   ├── isolation_forest.joblib
│   ├── scaler.joblib
│   └── feature_columns.json
│
├── logs/                                 # Log files
│
└── .venv/                                # Virtual environment (ignored)

9. Installation Guide

Prerequisites

Python 3.11 or later
pip (or uv for faster installs)

Setup

# Clone or extract the project
cd os-anomaly-sentinel

# Create virtual environment
python -m venv .venv

# Activate
source .venv/bin/activate      # Linux / macOS / Colab
# .venv\Scripts\activate       # Windows

# Install dependencies
pip install -r requirements.txt

Optional: Use uv Instead of pip

uv is significantly faster:

pip install uv
uv venv
source .venv/bin/activate
uv pip install -r requirements.txt

Virtual Environment Activation Summary

Platform	Command
Linux / macOS	`source .venv/bin/activate`
Windows (CMD)	`.venv\Scripts\activate`
Windows (PowerShell)	`.venv\Scripts\Activate.ps1`
Google Colab	`!python -m venv .venv && !source .venv/bin/activate && !pip install -r requirements.txt`

10. Quick Start Demo (Cross-Platform)

Run the complete pipeline with synthetic data (no real processes needed):

# Activate virtual environment first
source .venv/bin/activate

# 1. Generate synthetic data (30 min, 10 anomalies)
python run_agent.py --synthetic --duration 30

# 2. Train Isolation Forest baseline
python train_baseline.py --retrain

# 3. Run anomaly detection
python detect.py

# 4. Evaluate and compare 4 models
python evaluate.py

# 5. Generate final-year reports
python generate_report.py

# 6. Launch interactive dashboard
streamlit run dashboard.py

Expected output after each command:

Step	Key Output	What to Check
`run_agent.py`	`data/raw/process_metrics.csv`	~9000 rows, columns include `is_anomaly_ground_truth`
`train_baseline.py`	`models/isolation_forest.joblib`	~75 feature windows, 11 features
`detect.py`	`data/anomalies/anomaly_events.csv`	~10 anomalies with severity
`evaluate.py`	`data/reports/evaluation_results.csv`	4-model comparison table printed
`generate_report.py`	`data/reports/FINAL_YEAR_REPORT.md`	All reports generated listing
`dashboard.py`	http://localhost:8501	Full interactive dashboard

11. Full Workflow Guide

Step 1: Generate Synthetic Data

python run_agent.py --synthetic --duration 30 --anomalies 15

What it does: Generates labelled synthetic process telemetry without actual system monitoring. Outputs data/raw/process_metrics.csv.

Ground-truth columns (for evaluation only — never used for training):

is_anomaly_ground_truth — 1 if anomaly, 0 if normal
anomaly_type — which anomaly pattern was injected

Parameters:

Flag	Default	Description
`--synthetic`	—	Enable synthetic mode
`--duration`	5	Minutes of data to generate
`--interval`	2	Seconds between snapshots
`--anomalies`	10	Anomalies to inject

Step 2: Train Baseline Model

python train_baseline.py --retrain

What it does:

Loads raw telemetry from data/raw/process_metrics.csv
Engineers 11 statistical features over 5-minute windows
Trains Isolation Forest (unsupervised — no labels used)
Saves model artifacts to models/

Artifacts produced:

File	Purpose
`models/isolation_forest.joblib`	Trained Isolation Forest
`models/scaler.joblib`	StandardScaler for feature normalization
`models/feature_columns.json`	Feature names and order

Parameters:

Flag	Default	Description
`--retrain`	—	Force retrain (otherwise skips if model exists)
`--window`	5	Aggregation window in minutes
`--contamination`	0.1	Expected anomaly proportion

Step 3: Run Anomaly Detection

python detect.py

What it does: Scores each process window against the trained baseline. Outputs data/anomalies/anomaly_events.csv with severity, confidence, and explanations.

Detection columns: process_name, window_start, anomaly_score, is_anomaly, severity, confidence, category, top_features, explanation.

Parameters:

Flag	Default	Description
`--contamination`	0.1	Threshold for anomaly classification

Step 4: Evaluate Models

python evaluate.py

What it does: Compares 4 models against ground-truth labels:

Isolation Forest — main unsupervised detector
Fixed Threshold — rule-based baseline (CPU > 80 % or mem > 800 MB)
LOF — Local Outlier Factor (density-based comparison)
One-Class SVM — boundary-based comparison

Outputs to data/reports/:

File	Contents
`evaluation_results.csv`	Precision, recall, F1, accuracy, FP, FN, times
`evaluation_summary.json`	Best model, per-model metrics
`confusion_matrix.csv`	TN, FP, FN, TP
`model_comparison.md`	Formatted comparison report

Parameters:

Flag	Default	Description
`--contamination`	0.1	Expected anomaly proportion
`--cpu-threshold`	80.0	Fixed CPU % threshold
`--mem-threshold`	800.0	Fixed memory threshold (MB)

Step 5: Generate Reports

python generate_report.py

What it does: Generates 5 markdown reports from evaluation data:

File	Description
`data/reports/FINAL_YEAR_REPORT.md`	Complete 13-chapter final-year report
`data/reports/PROJECT_ABSTRACT.md`	One-page project abstract
`data/reports/viva_questions.md`	45+ likely viva Q&A pairs
`data/reports/technical_architecture.md`	Architecture and module documentation
`data/reports/demo_summary.md`	Step-by-step 10-step demo script

Step 6: Launch Dashboard

streamlit run dashboard.py

URL: http://localhost:8501

The dashboard auto-detects existing data, models, and evaluation results. No manual import needed.

12. Streamlit Frontend Guide

Page-by-Page Reference

#	Page	Purpose	Key Controls	When to Use
1	📊 Overview	System status, metric cards, pipeline health, architecture flow, CPU/memory trends, anomaly scores	Auto-loads from files	First page — shows project state
2	📡 Live Monitoring	Generate synthetic data or collect live psutil telemetry	Duration, Interval, Anomalies sliders; Generate button; Refresh button	At start — generate data or begin live collection
3	🧠 Baseline Training	Train/retrain Isolation Forest	Window, Contamination inputs; Train/Retrain buttons	After data generation
4	🔍 Anomaly Detection	Score features, filter results	Contamination input; Run Detection button; Process/Severity dropdowns	After training
5	🔎 Anomaly Investigation	Deep-dive into specific anomaly	Searchable data table with column controls	After detection — inspect individual anomalies
6	📈 Model Evaluation	Compare 4 models	Contamination, thresholds inputs; Run Full Evaluation button	After detection
7	📁 Data Explorer	Browse and export raw/processed/anomaly CSVs	3 tabs; filter by process; download buttons	Any time — inspect or export data
8	📄 Reports	View/download generated reports	Report selector dropdown; Download buttons	After report generation
9	⚙️ Settings	Edit all config.yaml parameters	Input fields per config section; Save/Reset buttons	As needed — adjust pipeline config
10	❓ Help / About	Documentation, methodology, viva context	6 expander sections	Reference — understand the project

Sidebar Status Bar

The sidebar shows real-time file state at the top:

Raw: ~9000 rows | Model: ✓ | Eval: ✓ | Anomalies: ~10

This updates automatically when files change (generation, training, detection, evaluation from CLI or UI).

13. ML Methodology

Models Implemented

Model	Type	Role
Isolation Forest	Unsupervised (main)	Default detection — learns normal behaviour via recursive partitioning
Fixed Threshold	Rule-based (baseline)	Comparison only — CPU > 80 % or memory > 800 MB
Local Outlier Factor	Unsupervised (comparison)	Density-based neighbourhood analysis
One-Class SVM	Unsupervised (comparison)	Boundary-based novelty detection

Feature Engineering

Each 5-minute process window produces 11 features:

Feature	Description
`avg_cpu`	Mean CPU % across snapshots
`max_cpu`	Peak CPU %
`std_cpu`	CPU variability
`avg_mem_mb`	Mean memory (MB)
`max_mem_mb`	Peak memory (MB)
`std_mem_mb`	Memory variability
`avg_mem_percent`	Mean memory %
`max_mem_percent`	Peak memory %
`process_seen_count`	Number of snapshots in window
`cpu_trend`	Linear CPU trend coefficient
`mem_trend`	Linear memory trend coefficient

Evaluation Metrics

Precision: proportion of detected anomalies that are true positives
Recall: proportion of true anomalies that were detected
F1-score: harmonic mean of precision and recall
Accuracy: overall correct classification rate
False Positives (FP): normal windows incorrectly flagged
False Negatives (FN): missed anomalies
Confusion Matrix: TN, FP, FN, TP per model
Training Time: seconds to train
Inference Time: seconds to score

14. Synthetic Dataset Guide

Why Synthetic Data?

Synthetic data enables cross-platform demos without psutil access (Colab, remote servers) and provides controlled ground-truth labels for evaluation.

Important: Labels Are for Evaluation Only

The synthetic generator produces is_anomaly_ground_truth and anomaly_type columns, but these are never used during training. Isolation Forest, LOF, and One-Class SVM are unsupervised — they learn without labels. Labels are only used in evaluate.py to measure how well each unsupervised model detects known anomalies.

Anomaly Types Injected

Type	Pattern
`cpu_spike`	Sudden CPU % surge in one process
`memory_leak`	Gradual memory growth over time
`resource_hog`	Simultaneous CPU + memory increase
`process_burst`	Abnormal process spawn count
`silent_drift`	Slow, subtle metric drift
`rare_behavior`	Unusual process running briefly

Generating with Custom Parameters

# 60 minutes, 5-second intervals, 20 anomalies
python run_agent.py --synthetic --duration 60 --interval 5 --anomalies 20

15. Real Live Monitoring Guide

# Collect live psutil telemetry for 10 minutes
python run_agent.py --duration 10

Platform notes:

Works on Linux, Windows, and macOS
Some protected system processes return psutil.AccessDenied — this is handled safely (logged, not crashed)
Administrator/root privileges may expose more processes but are not required
No process is killed, modified, or suspended — read-only monitoring only

After live collection:

python train_baseline.py --retrain
python detect.py
streamlit run dashboard.py

16. Jupyter Notebook Guide

The project includes OS_Anomaly_Detection_Project.ipynb with 19 sections covering the full pipeline, architecture, results, and viva preparation.

Usage

Open in VS Code, Jupyter Lab, or Google Colab:

jupyter notebook OS_Anomaly_Detection_Project.ipynb

Validate Notebook Structure

python tests/validate_notebook.py

Expected output:

Notebook: OS_Anomaly_Detection_Project.ipynb
  nbformat: v4.5
  Cells: 54 total (19 markdown, 35 code, 0 raw, 0 other)
Result: VALID (0 warnings)

17. Manual Visual UI Verification

The Streamlit frontend was manually verified through visual inspection and user testing. Each dashboard page was opened, checked, and captured as a screenshot.

Verified Pages

#	Page	Key Controls Tested
1	Overview	Metric cards, pipeline health, charts
2	Live Monitoring	Duration/Interval/Anomalies sliders, Generate button
3	Baseline Training	Window/Contamination inputs, Train/Retrain buttons
4	Anomaly Detection	Contamination input, Run Detection, filter dropdowns
5	Anomaly Investigation	Data table, column controls, search
6	Model Evaluation	Threshold inputs, Run Full Evaluation, metric charts
7	Data Explorer	3 data tabs, process filter, download buttons
8	Reports	Report selector dropdown, download buttons
9	Settings	Config inputs, Save/Reset buttons
10	Help / About	6 expander sections, safety documentation

Manual Checks Performed

Page navigation across all 10 pages
Metric cards displaying correct values
Charts rendering with Plotly (zoom, pan, download controls)
Tables with sort, filter, search, and download
Filter dropdowns and comboboxes
Generate/Train/Detect/Evaluate buttons
Report previews and download buttons
Settings save and reset flow
Missing-file edge case handling (no data, no model)
Safety messaging and ethics documentation

Screenshots

All 10 page screenshots are stored in data/reports/screenshots/:

#	File	Page
1	`01_overview.png`	Overview
2	`02_live_monitoring.png`	Live Monitoring
3	`03_baseline_training.png`	Baseline Training
4	`04_anomaly_detection.png`	Anomaly Detection
5	`05_anomaly_investigation.png`	Anomaly Investigation
6	`06_model_evaluation.png`	Model Evaluation
7	`07_data_explorer.png`	Data Explorer
8	`08_reports.png`	Reports
9	`09_settings.png`	Settings
10	`10_help_about.png`	Help / About

Manual Verification Report

data/reports/MANUAL_UI_VERIFICATION_REPORT.md — full QA report with per-page checklists, acceptance criteria, and bug/fix log.

18. Testing Guide

Unit Tests (26 tests, 5 files)

# Run all unit tests
python -m unittest discover tests/ -v

Test files:

File	What It Tests
`tests/test_synthetic.py`	Synthetic data generation, anomaly injection, ground-truth columns, edge cases (0 anomalies)
`tests/test_features.py`	Feature engineering, window aggregation, trend calculation
`tests/test_model.py`	Model training, artifact saving/loading, prediction shape
`tests/test_detector.py`	Detection pipeline, severity classification, explanation generation
`tests/test_evaluation.py`	Metric calculation, confusion matrix, model comparison

Compile Check

python -c "
import glob, py_compile
files = ['dashboard.py','run_agent.py','train_baseline.py','detect.py',
         'evaluate.py','generate_report.py','package_app.py']
files += sorted(glob.glob('src/*.py') + glob.glob('src/ui/*.py') +
                glob.glob('tests/*.py'))
for f in files:
    py_compile.compile(f, doraise=True)
print(f'ALL {len(files)} FILES COMPILE OK')
"

19. Packaging Guide

CLI Scripts as Standalone Executables

# Agent
pyinstaller --onefile --name OSAnomalyAgent run_agent.py

# Detector
pyinstaller --onefile --name OSAnomalyDetector detect.py

# Evaluator
pyinstaller --onefile --name OADEvaluator evaluate.py

Streamlit Dashboard

Streamlit apps are web servers. Packaging with PyInstaller is complex and often brittle. The recommended approach:

pip install -r requirements.txt
streamlit run dashboard.py

For a launcher script or Docker deployment, see package_app.py.

Windows Deployment (No Python Required)

For end users without Python:

Install Python 3.11+ from python.org (check "Add to PATH")
Open Command Prompt in the project folder
python -m venv .venv && .venv\Scripts\activate
pip install -r requirements.txt
Run synthetic demo commands (Section 10)
streamlit run dashboard.py

20. Final Submission Zip Guide

zip -r OS_Anomaly_Sentinel_Final_Year_Project.zip \
  README.md \
  config.yaml \
  requirements.txt \
  OS_Anomaly_Detection_Project.ipynb \
  dashboard.py \
  run_agent.py \
  train_baseline.py \
  detect.py \
  evaluate.py \
  generate_report.py \
  package_app.py \
  src/ \
  tests/ \
  data/reports/FINAL_YEAR_REPORT.md \
  data/reports/PROJECT_ABSTRACT.md \
  data/reports/viva_questions.md \
  data/reports/technical_architecture.md \
  data/reports/demo_summary.md \
  data/reports/model_comparison.md \
  data/reports/MANUAL_UI_VERIFICATION_REPORT.md

Do not include (unless required by your evaluator):

data/raw/ — regenerated with run_agent.py
data/processed/ — regenerated with train_baseline.py
data/anomalies/ — regenerated with detect.py
models/ — regenerated with train_baseline.py
.venv/ — too large; always recreate
logs/ — session-specific
__pycache__/ — auto-generated

21. Recommended Final Demo Flow

Open README.md — explain project objective and architecture
Generate synthetic data — python run_agent.py --synthetic --duration 30 --anomalies 15
Train model — python train_baseline.py --retrain
Run detection — python detect.py
Run evaluation — python evaluate.py
Generate reports — python generate_report.py
Launch dashboard — streamlit run dashboard.py
Show Overview page — pipeline health, charts, metric cards
Show Model Evaluation page — 4-model comparison, confusion matrix
Show Anomaly Investigation — deep-dive explanation
Show Reports page — final-year report, viva questions
Show tests passing — python -m unittest discover tests/ -v
Show manual UI verification report — screenshot folder, UI verification report

22. Safety and Ethics

No process termination: The system flags anomalies for human investigation only
No kernel modifications: All monitoring uses user-space psutil
No labelled data required: Unsupervised learning avoids privacy concerns around manual labelling
Not a security tool: Anomaly means "statistical deviation," not "confirmed threat"
Transparency: All detections include human-readable explanations with confidence levels
AccessDenied handling: Protected processes are logged and skipped, not crashed

23. Limitations

Cold start: Model needs representative training data before detection is meaningful
Workload drift: Permanent workload changes require retraining
Protected processes: Some system processes inaccessible to psutil due to OS permissions
Single-machine scope: Currently monitors one machine at a time
Synthetic evaluation: Real-world evaluation depends on expert labelling
No online learning: Retraining is batch-based, not incremental
Streamlit is a demo frontend: Not designed for enterprise deployment or production alerting

24. Future Scope

Online/incremental learning for continuous model updates
Multi-machine aggregation dashboard for fleet monitoring
Autoencoder-based deep anomaly detection (PyTorch)
Alert integration (webhook, Slack, email)
SHAP/LIME integration for enhanced explainability
Time-series forecasting for predictive anomaly detection
Windows service / Linux daemon mode
Signed installer for end-user deployment

25. Troubleshooting

Issue	Solution
`ModuleNotFoundError`	Run `pip install -r requirements.txt`
`No raw data found`	Run `python run_agent.py --synthetic --duration 30`
`No model found`	Run `python train_baseline.py --retrain`
`No features found`	Training automatically runs feature engineering
`Streamlit not found`	Run `pip install streamlit`
Dashboard empty / no data	Generate data and train model via CLI or UI buttons
`psutil.AccessDenied`	Some system processes are protected; this is expected and handled
Notebook validation warning	Cosmetic `language_info` warning — does not affect execution
`config.yaml` path issues	Ensure `config.yaml` is in the project root
Evaluation says no ground truth	Regenerate with current synthetic module: `run_agent.py --synthetic --duration 30`
`pyinstaller` not found	`pip install pyinstaller`
`uv` command not found	Use `pip` instead, or `pip install uv` first

26. Viva Questions Preview

The project includes 45+ viva preparation questions with detailed answers in data/reports/viva_questions.md. Below are 10 key questions:

#	Question	Short Answer
1	Why use Isolation Forest over fixed thresholds?	IF adapts per-machine; thresholds are static and miss drifts or generate FPs
2	How does feature engineering work?	5-min windows aggregated to 11 stats: avg, max, std, trend for CPU/mem + count
3	Why unsupervised and not supervised?	Labelled OS anomaly data is rare, expensive, and privacy-sensitive
4	What do precision/recall/F1 mean here?	Precision = how many flagged are real; Recall = how many real are flagged; F1 = balance
5	How are false positives handled?	FPs are expected from unsupervised methods; human investigation is required
6	Why compare with Fixed Threshold?	To prove adaptive ML outperforms the industry-standard rule-based approach
7	What are the limitations?	Cold start, workload drift, single-machine, no online learning, synthetic eval
8	How would you scale this?	Multi-machine stream processing, incremental learning, alert webhooks
9	Is this a security tool?	No — anomaly = statistical deviation, requiring human investigation
10	What would you improve?	Online learning, SHAP explainability, multi-host fleet view, alert integrations

27. Clarification: Fixed Thresholds Are a Baseline, Not the Main Method

The main anomaly detector (Isolation Forest) does not rely on hard-coded thresholds. Fixed CPU/memory thresholds are included only as a baseline for comparison during evaluation.

This is a common point of confusion. The project's entire motivation is that fixed thresholds fail. Isolation Forest adapts to each machine's normal — no manual tuning, no static rules.

28. Generated Report Files Summary

After running the full pipeline, data/reports/ contains:

File	Generator	Description
`FINAL_YEAR_REPORT.md`	`generate_report.py`	13-chapter project report for submission
`PROJECT_ABSTRACT.md`	`generate_report.py`	One-page abstract for evaluators
`viva_questions.md`	`generate_report.py`	45+ likely questions with detailed answers
`technical_architecture.md`	`generate_report.py`	Architecture deep-dive
`demo_summary.md`	`generate_report.py`	10-step demo walkthrough
`evaluation_results.csv`	`evaluate.py`	Per-model metrics table
`evaluation_summary.json`	`evaluate.py`	Programmatic evaluation results
`confusion_matrix.csv`	`evaluate.py`	TN, FP, FN, TP matrix
`model_comparison.md`	`evaluate.py`	Formatted comparison report
`MANUAL_UI_VERIFICATION_REPORT.md`	Manual	Full manual visual QA report (PASS)

OS Anomaly Sentinel v2.0 — Final Year Project Built with Python, scikit-learn, Streamlit, and psutil

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
data		data
logs		logs
models		models
src		src
tests		tests
OS_Anomaly_Detection_Project.ipynb		OS_Anomaly_Detection_Project.ipynb
README.md		README.md
config.yaml		config.yaml
dashboard.py		dashboard.py
detect.py		detect.py
evaluate.py		evaluate.py
generate_report.py		generate_report.py
package_app.py		package_app.py
requirements.txt		requirements.txt
run_agent.py		run_agent.py
train_baseline.py		train_baseline.py

Folders and files

Latest commit

History

Repository files navigation

OS Anomaly Sentinel

1. Project Overview

2. Problem Statement

3. Proposed Solution

4. Key Features

5. Final-Year Project Highlights

6. System Architecture

Data Flow

Module-Level Flow

7. Tech Stack

8. Repository Structure

9. Installation Guide

Prerequisites

Setup

Optional: Use uv Instead of pip

Virtual Environment Activation Summary

10. Quick Start Demo (Cross-Platform)

11. Full Workflow Guide

Step 1: Generate Synthetic Data

Step 2: Train Baseline Model

Step 3: Run Anomaly Detection

Step 4: Evaluate Models

Step 5: Generate Reports

Step 6: Launch Dashboard

12. Streamlit Frontend Guide

Page-by-Page Reference

Sidebar Status Bar

13. ML Methodology

Models Implemented

Feature Engineering

Evaluation Metrics

14. Synthetic Dataset Guide

Why Synthetic Data?

Important: Labels Are for Evaluation Only

Anomaly Types Injected

Generating with Custom Parameters

15. Real Live Monitoring Guide

16. Jupyter Notebook Guide

Usage

Validate Notebook Structure

17. Manual Visual UI Verification

Verified Pages

Manual Checks Performed

Screenshots

Manual Verification Report

18. Testing Guide

Unit Tests (26 tests, 5 files)

Compile Check

19. Packaging Guide

CLI Scripts as Standalone Executables

Streamlit Dashboard

Windows Deployment (No Python Required)

20. Final Submission Zip Guide

21. Recommended Final Demo Flow

22. Safety and Ethics

23. Limitations

24. Future Scope

25. Troubleshooting

26. Viva Questions Preview

27. Clarification: Fixed Thresholds Are a Baseline, Not the Main Method

28. Generated Report Files Summary

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages