Network Intrusion Detection — KDD Cup 99

A complete data-mining pipeline that classifies network connections as Normal or Attack using the 10%-subset of the classic KDD Cup 1999 intrusion-detection dataset. Built as a course project for a data mining class, covering the full workflow from raw data to a tuned, evaluated classifier — EDA, cleaning, preprocessing, dimensionality reduction, clustering, and supervised model comparison.

Results

Model	Accuracy	Precision (Attack)	Recall (Attack)	F1 (Attack)	ROC-AUC
Dummy Baseline	0.6033	0.0000	0.0000	0.0000	0.5000
Logistic Regression	0.9956	0.9939	0.9950	0.9945	0.9998
Logistic Regression + SVD	0.9868	0.9825	0.9842	0.9833	0.9993
Random Forest	0.9991	0.9997	0.9982	0.9989	1.0000

Random Forest was selected as the final model based on attack-class F1. On the held-out test set it misses only 21 attacks and misclassifies just 4 normal connections as attacks (out of 29,118 test records).

Dataset summary

• 494,021 raw records, 41 original features, 0 missing values
• 348,435 exact duplicates removed → 145,586 clean records (87,832 normal / 57,754 attack)
• Two constant columns dropped (num_outbound_cmds, is_host_login)
• 80/20 stratified train/test split → 116,468 train / 29,118 test records

Unsupervised analysis

• Truncated SVD to 20 components retains 98.66% of variance
• MiniBatch K-Means clustering vs. true labels: Adjusted Rand Index = 0.8674, silhouette score = 0.4256

Pipeline

The full implementation walks through 12 stages:

1. Understand the dataset structure and feature types from the metadata file
2. Check for missing values, duplicate records, and constant columns
3. Remove exact duplicates before the train/test split (to avoid leakage-like evaluation)
4. Exploratory data analysis with saved plots
5. Impute missing values inside the pipeline (for robustness, even though this dataset has none)
6. Clip numerical outliers via quantile clipping, fitted only on training data
7. One-hot encode categorical features and standard-scale numerical features
8. Stratified 80/20 train/test split
9. Dimensionality reduction with Truncated SVD; compare model performance with and without it
10. MiniBatch K-Means clustering, compared against true labels
11. Compare a Dummy Baseline, Logistic Regression, and Random Forest
12. Evaluate with accuracy, precision, recall, F1, ROC-AUC, confusion matrix, and ROC curve

All code comments are in English. Running the script (or notebook) regenerates outputs/ — plots, metrics, and reports — locally; summary_report.md is a saved copy of the execution summary from a real run, so results can be reviewed without re-running anything.

Project structure

project/
├── intrusion_detection_project.py     # Full, runnable pipeline
├── README.md
└── requirements.txt                   # Python dependencies

Getting started

Place the dataset archive 4.zip (the 10%-subset of KDD Cup 99) next to the Python script — the script reads it directly without manual extraction — then run:

pip install -r requirements.txt
python intrusion_detection_project.py --zip-path 4.zip --output-dir outputs

On Google Colab, upload 4.zip and the script, and run the same command with !python.

License

Educational project — no license specified. Contact the author before reuse.