title	Trend Engine
emoji	🌐
colorFrom	blue
colorTo	indigo
sdk	streamlit
app_file	src/dashboard/app.py
pinned	false

🌐 Trend Discovery Engine

An end-to-end Machine Learning and Natural Language Processing system for discovering, analyzing, and projecting emerging technology trends from unstructured online discussions.

---

📖 Project Overview

The Trend Discovery Engine collects, processes, and structures technology discussions from Hacker News. By transforming unstructured titles and comments into high-dimensional semantic embeddings, it applies density-based clustering to uncover topic communities. It then performs temporal analysis on these communities to identify and score emerging technology trends over time, presenting them in an interactive dashboard.

This platform automatically surfaces active technical domains, such as AI Agents, Agentic Security, Vector Databases, and LLM Infrastructure, before they become mainstream.

---

🛠️ Tech Stack & Key Libraries

Component	Technology / Library	Description
Data Ingestion	Algolia Hacker News Search API	Scrapes front page and targeted technical discussions
NLP & Embeddings	sentence-transformers (all-MiniLM-L6-v2)	Embeds posts into 384-dimensional dense vectors
Dimension Reduction	umap-learn	Project embeddings to 2D for visualization and 20D for clustering
Clustering	hdbscan	Discovers dense semantic topics without specifying k pre-emptively
Interpretation	scikit-learn (TF-IDF)	Generates class-based TF-IDF topic keywords for discovered clusters
Analysis	pandas & numpy	Computes rolling growth rates and weighted trend scores
Dashboard	streamlit & plotly	Renders a premium interactive technology intelligence platform

---

🧬 Machine Learning Pipeline Architecture

            [ Hacker News Discussions ]
                        │
                        ▼
               [ Data Ingestion ]
           (Targeted queries & Front Page)
                        │
                        ▼
         [ Data Cleaning & Preprocessing ]
          (Duplicate removals, Length checks)
                        │
                        ▼
      [ Sentence Transformer Embedding (384D) ]
                        │
                        ▼
       [ UMAP Dimensionality Reduction (20D) ]
                        │
                        ▼
          [ HDBSCAN Density Clustering ]
                        │
            ┌───────────┴───────────┐
            ▼                       ▼
   [ Topic Discovery ]      [ Trend Detection ]
    (Class-based TF-IDF)     (Rolling growth & scoring)
            │                       │
            └───────────┬───────────┘
                        ▼
         [ Streamlit Web Intelligence ]

---

📁 Project Directory Structure

trend-discovery-engine/
├── src/
│   ├── scrapers/          # Scraper classes and execution scripts
│   │   ├── hn_scraper.py
│   │   └── run_hn_scraper.py
│   ├── preprocessing/     # Cleaners and preprocessing pipelines
│   │   ├── hn_cleaner.py
│   │   └── run_preprocessing.py
│   ├── embeddings/        # Embedding generators and UMAP reductions
│   │   ├── embedder.py
│   │   ├── run_embeddings.py
│   │   └── run_umap.py
│   ├── clustering/        # Clustering pipelines and TF-IDF interpreters
│   │   ├── hdbscan_clusterer.py
│   │   ├── cluster_inspector.py
│   │   ├── cluster_interpreter.py
│   │   ├── run_clustering.py
│   │   ├── run_inspection.py
│   │   └── run_interpretation.py
│   ├── trends/            # Growth analysis and score engines
│   │   ├── trend_analyzer.py
│   │   └── run_trend_analysis.py
│   ├── forecasting/       # Trend activity regression and model benchmarking
│   │   ├── feature_engineering.py
│   │   ├── run_feature_engineering.py
│   │   ├── train_regression.py
│   │   ├── predict_future_trends.py
│   │   └── benchmark_models.py
│   ├── classification/    # Imbalanced trend growth classification
│   │   └── train_growth_classifier.py
│   └── dashboard/         # Streamlit UI page components and loaders
│       ├── app.py
│       ├── pages/
│       │   ├── overview.py
│       │   ├── trend_explorer.py
│       │   ├── cluster_explorer.py
│       │   └── umap_vis.py
│       └── utils/
│           └── data_loader.py
├── tests/                 # Full unit testing suite
├── data/                  # Raw, processed, and pipeline artifacts
└── README.md

---

Results & Performance

Unsupervised Learning Performance

The semantic clustering pipeline was evaluated using multiple embedding models and clustering configurations.

Embedding Model	Silhouette Score	Davies-Bouldin (lower is better)	Calinski-Harabasz (higher is better)	Clusters	Noise %
all-MiniLM-L6-v2	0.0584	2.5104	9.1259	85	34.04%
bge-small-en-v1.5	0.0467	2.5639	11.4340	71	29.04%

After evaluating multiple UMAP and HDBSCAN configurations, the default pipeline configuration was retained as it provided the best balance between topic granularity and semantic coherence.

Forecasting Model Benchmark

Three forecasting approaches were benchmarked using 5-fold TimeSeriesSplit Cross-Validation.

Rank	Model	MAE (lower is better)	RMSE (lower is better)	R2 (higher is better)
1	Linear Regression	0.3919	1.2305	0.0604
2	XGBoost Regressor	0.4463	1.3010	-0.4855
3	Random Forest Regressor	0.4542	1.2958	-0.5710

Interestingly, the simpler Linear Regression model outperformed ensemble-based methods, indicating that short-term technology trend dynamics exhibit largely linear behavior.

Trend Growth Classification

The platform also predicts whether a technology trend is likely to grow in the following week.

Models were evaluated using 5-fold TimeSeriesSplit on a highly imbalanced dataset (~9.3% positive growth examples).

Model	Precision	Recall	F1 Score	ROC-AUC
RandomForestClassifier	0.2021	0.0528	0.0515	0.6304
XGBClassifier	0.1584	0.1561	0.1204	0.6163

The XGBoost classifier was selected as the final production model due to its superior recall and F1-score, making it more effective at identifying emerging technology trends.

Example Emerging Trends Discovered

The pipeline successfully identified and tracked emerging ecosystems including:

• AI Agents & Agentic Workflows
• Agent Security & Prompt Injection Defense
• Open-Weight LLM Ecosystems (DeepSeek, GLM)
• Vector Databases & Retrieval Infrastructure
• Agentic Memory Architectures
• AI Coding Agents & Developer Tooling

---

🚀 Getting Started

1. Installation

Clone the repository and set up your virtual environment:

# Clone repository
git clone <repository-url>
cd trend-discovery-engine

# Initialize environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirement.txt

2. Running the ML Pipeline Step-by-Step

Execute each pipeline module sequentially to generate the workspace artifacts:

Note

Ensure you run these scripts in order, as each step depends on the output of the previous step.

# Step 1: Collect raw Hacker News data
python src/scrapers/run_hn_scraper.py --limit 50 --front-page

# Step 2: Preprocess and clean the raw dataset
python src/preprocessing/run_preprocessing.py

# Step 3: Generate dense semantic embeddings
python src/embeddings/run_embeddings.py

# Step 4: Run UMAP dimensionality reduction
python src/embeddings/run_umap.py

# Step 5: Run HDBSCAN clustering
python src/clustering/run_clustering.py

# Step 6: Interpret clusters using c-TF-IDF
python src/clustering/run_interpretation.py

# Step 7: Analyze temporal growth and score emerging trends
python src/trends/run_trend_analysis.py

# Step 8: Build historical dataset with lag & acceleration features per cluster
python src/forecasting/run_feature_engineering.py

# Step 9: Benchmark different regression forecasting models using 5-fold TimeSeriesSplit CV
python src/forecasting/benchmark_models.py

# Step 10: Train final forecasting model (using Linear Regression, which outperformed XGBoost & RF in benchmarks)
python src/forecasting/train_regression.py

# Step 11: Project next week's trend activity (predicted posts counts)
python src/forecasting/predict_future_trends.py

# Step 12: Train and compare imbalanced trend growth classifiers (RandomForest vs XGBoost with imbalance weightings)
python src/classification/train_growth_classifier.py

3. Launching the Interactive Dashboard

Launch the Streamlit dashboard to explore the discovered trends and visualize the technology landscape:

streamlit run src/dashboard/app.py

Open your browser and navigate to http://localhost:8501.

---

📊 Dashboard Modules

• 🌐 Technology Overview: High-level metrics showing total stories, identified clusters, and noise percentage, along with a ranked plot of the top 10 trends.
• 📈 Trend Explorer: Complete, searchable technology trends table sorted by trend score, supporting real-time keyword filtering.
• 🔍 Cluster Inspector: Deep-dive details on any specific topic, displaying its timeline, growth rate, and top 10 representative articles sorted by score.
• 🗺️ Interactive UMAP Visualizer: A Plotly scatter plot of the embedding space. Active clusters are highlighted in distinct colors while noise points are colored grey with low opacity (0.3) for clean visualization.

🧪 Running Unit Tests

Run the full testing suite to verify system integrity:

python -m unittest discover -s tests

---

📄 License

This project is licensed under the MIT License.