SQLite-Database Integrated Log Analysis.

A multi-format approach full-stack desktop application which analyzes log files and detects patterns within them. The project utilizes an SQLite database loader and a Tkinter-based UI as well as a separate log-generator file to test the project with sample logs.

Overview

Developed a comprehensive SQLite-integrated log analysis system utilizing python architectures throguh regex to provide real-time pattern detection and database persistence. The application employs a tkinter UI design which relies on separated modules: config.py, databse_manager.py, pattern_detector.py, log_analyzer.py, ui_components.py, and main.py to build the interface.

The core pattern detection engine employs compiled regex patterns using case-insensitive matching to find critical log patterns with three severity levels: HIGH (correlates to exceptions, errors, and large failures in inputted log files), MEDIUM (resulting in warnings and HTTP 4xx/5xx codes), and INFO (correlating with timestamps, IP addresses, URLs and JSON Objects and isn't representative of mitigating severity of threats in log files). The system further accounts for multiple encoding variations involving UTF-8, Latin-1, and CP1252 to ensure compatibility with log formats. Databse integration utilizes SQLite with normalized tables involving pattern_specific results, user_defined regex queries and analysis metadata. Export functionality generates JSON reports with complete analysis data as well.

Separate from the main user interface is a log generator file (generate_sample_logs.py) which creates test datasets across distinct log formats, involving application logs with auth. events and system errors, web server logs that utilize the common Apache/Nginx log format, system logs that model Linux entries with kernel messages and service events, and API logs (JSON-formatted) with request/response data. Overall the generator can produce over 5,000 instances of realistic log entries spanning several days of simulated activity (through randomized timestamps) to test pattern detection and analysis capabilities across varying log formats.

Getting Started

Prerequisites

• Python 3.7 or higher.
• No additional packages required (uses built-in libraries).

Installation & Running (Mac + Windows)

Windows:

# Clone or download the project -> provide directory for the file on your computer
cd smart-log-analyzer

# Run the application
python main.py

Mac/Linux:

# Clone or download the project -> provide directory for the file on your computer
cd smart-log-analyzer

# Run the application
python3 main.py

File Structure

File	Purpose
main.py	Main application entry point where the GUI is built and other files are all called.
config.py	Configuration settings which include UI colors, fonts, and regex patterns.
database_manager.py	SQLite database operations accounting for database functionality.
log_analyzer.py	Core analysis logic combining pattern detection with database storage.
pattern_detector.py	Regex-based pattern detection engine with severity classification.
ui_components.py	Custom UI components, styles, and widget factories.
generate_sample_logs.py	Utility to create test sample log files to demonstrate project.

Generating Sample Logs

To create test log files for the project:

# Run the log generator
python generate_sample_logs.py

This creates a sample_logs/ directory with 4 different log types:

• application.log - 2000 entries with errors, warnings, user activities.
• webserver.log - 1500 web server access logs (Apache format).
• system.log - 1000 system events (Linux syslog format).
• api.log - 800 JSON-formatted API request logs.

Detailed Explanation on How to Use

1. Launch Application: Run python main.py
2. Load Log File: Click "Load Log File" and select log file of choices.
3. Analyze Patterns: Click "Analyze Patterns" (will auto-run after loading).
4. Explore Results: Use the 4 tabs to view:
   • Analysis Results: Summary and detected patterns.
   • Pattern Search: Custom regex pattern searches.
   • Raw Logs: Searchable content in the raw logs.
   • Statistics: Database stats and session history.
5. Export Data: Click "Export JSON Report" to save results.

Pattern Detection

The system automatically detects:

• HIGH Severity: Errors, exceptions, failures, fatal events.
• MEDIUM Severity: Warnings, HTTP 4xx/5xx codes, deprecated functions.
• INFO Severity: IP addresses, timestamps, URLs, JSON objects, emails.

Database Storage

All analysis sessions will be stored in log_analysis.db with:

• Session metadata (filename, timestamp, line counts).
• Detected patterns with match counts and samples.
• Custom search history.

Application Interface

The application features a Dark Tkinter-based GUI with:

• Tabbed interface for viewing data,
• Sortable tables with pattern results,
• Syntax highlighting for log content.
• Real-time search in raw logs and for regex patterns.

Technical Features

Skills involved in project:

• Object-oriented Python architecture.
• GUI development with Tkinter.
• Database design and SQLite integration.
• Regex pattern matching and text processing.
• Threading for responsive user interfaces.

License

Open source project - feel free to use and modify as needed.