Skip to content

README.md

Latest commit

 

History

History
360 lines (273 loc) · 12.5 KB

README.md

File metadata and controls

360 lines (273 loc) · 12.5 KB
Raw

Tweet Sentiment & Emotion Analysis — Scalable Big Data Pipeline

A distributed, real-time tweet analytics system that simultaneously performs binary sentiment classification (positive / negative) and six-class emotion recognition (sadness, joy, love, anger, fear, surprise) on large-scale Twitter data. The pipeline is end-to-end: raw CSVs on HDFS → Spark batch preprocessing → MLlib model training → live Kafka stream inference → MongoDB persistence → Streamlit dashboard.

Table of Contents

Overview

The system is designed to process tweet-scale data (tens of millions of records) using a fully distributed stack. Batch preprocessing and model training are handled by Apache Spark jobs running on a Spark standalone cluster backed by HDFS. Trained models are serialised as Spark MLlib PipelineModel objects and stored on HDFS. At inference time, Apache Spark Structured Streaming consumes a Kafka topic as micro-batches, applies both models concurrently, and writes dual-labelled predictions to MongoDB. A Streamlit dashboard reads from MongoDB in real time to display live results and allows analysts to inject arbitrary tweet text via an embedded Kafka producer.

Architecture

Raw CSVs (HDFS)
      │
      ▼
┌─────────────────────────────────────────────┐
│  Spark Batch Preprocessing  (Scala / SBT)   │
│  • URL / mention / hashtag removal          │
│  • Emoji transliteration (emoji-java)       │
│  • Lowercasing, punctuation stripping       │
│  → Parquet output back to HDFS              │
└─────────────────────────────────────────────┘
      │
      ▼
┌─────────────────────────────────────────────┐
│  Spark MLlib Model Training  (PySpark)      │
│  • Sentiment: TF-IDF + Bigrams → LR        │
│  • Emotion:   Bigram TF-IDF  → OvR + SVC   │
│  → PipelineModel serialised to HDFS         │
└─────────────────────────────────────────────┘
      │
      ▼
┌─────────────────────────────────────────────┐
│  Apache Kafka  (topic: tweetanalyse)        │
│  JSON messages: {"text": "..."}             │
└─────────────────────────────────────────────┘
      │
      ▼
┌─────────────────────────────────────────────┐
│  Spark Structured Streaming Inference       │
│  • Loads both PipelineModels from HDFS      │
│  • Applies sentiment + emotion models       │
│  • Merges dual labels per record            │
│  → Writes to MongoDB (foreachBatch)         │
└─────────────────────────────────────────────┘
      │
      ▼
┌─────────────────────────────────────────────┐
│  Streamlit Dashboard                        │
│  • Live sentiment / emotion charts          │
│  • Embedded Kafka producer (tweet input)    │
└─────────────────────────────────────────────┘

Tech Stack

Component Technology
Distributed compute Apache Spark 3.5.1 (standalone cluster)
Batch preprocessing Scala 2.12.18, SBT, emoji-java 5.1.1
ML training & inference PySpark MLlib (Logistic Regression, LinearSVC, OvR)
Distributed storage Hadoop HDFS 3.x (NameNode localhost:9000)
Message broker Apache Kafka (broker localhost:9092)
Database MongoDB (localhost:27017)
Dashboard Streamlit + Plotly

Repository Structure

.
├── src/
│   └── main/scala/
│       ├── TweetPreprocessor.scala        # Spark batch job — sentiment data cleaning
│       └── EmotionDataPreprocessor.scala  # Spark batch job — emotion data cleaning
├── sentiment_LR.py                        # PySpark — train Logistic Regression (sentiment)
├── emotion_svc.py                         # PySpark — train OvR LinearSVC (emotion)
├── streaming_dual_prediction.py           # PySpark Structured Streaming — live inference
├── dashboard.py                           # Streamlit dashboard
├── build.sbt                              # SBT build definition
├── run.txt                                # Quick-reference run commands
└── README.md

Datasets

Dataset Records Classes Description
Sentiment140 ~13.5 M 2 (pos / neg) Tweets with distant-supervision binary labels
CARER Emotion ~20 K 6 (sadness, joy, love, anger, fear, surprise) Fine-grained emotion text dataset

Place raw CSV files on HDFS before running the pipeline (see HDFS Paths).

Prerequisites

Ensure the following are installed and running before executing any pipeline step:

  • Java 8 or 11 (compatible with Spark 3.5.x)
  • Scala 2.12 + SBT
  • Apache Spark 3.5.1 — standalone master expected at spark://localhost:7077
  • Hadoop HDFS — NameNode expected at hdfs://localhost:9000
  • Python 3.8+ with the packages below: 
    pyspark==3.5.1
pymongo
streamlit
plotly
kafka-python
  • Apache Kafka — broker expected at localhost:9092; create topic before streaming: 
    kafka-topics.sh --create --topic tweetanalyse --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
  • MongoDB — expected at mongodb://127.0.0.1:27017

Setup & Configuration

All service endpoints are hard-coded in the respective scripts. If your environment differs, update the following:

Setting Default File(s)
Spark master spark://localhost:7077 sentiment_LR.py, emotion_svc.py, streaming_dual_prediction.py
HDFS NameNode hdfs://localhost:9000 All Scala & Python files
Kafka broker localhost:9092 streaming_dual_prediction.py, dashboard.py
Kafka topic tweetanalyse streaming_dual_prediction.py, dashboard.py
MongoDB URI mongodb://127.0.0.1:27017 streaming_dual_prediction.py, dashboard.py
MongoDB DB / collection tweetanalysis / predictions streaming_dual_prediction.py, dashboard.py

Running the Pipeline

Execute the steps in order. Steps 1–5 are offline (batch); steps 6–7 are online (streaming).

Step 1 — Build the Scala JAR

sbt package

Output: target/scala-2.12/tweet-sentiment_2.12-1.0.jar

Step 2 — Preprocess Emotion Data

Cleans the CARER emotion CSV on HDFS and writes Parquet output.

spark-submit \
  --class EmotionDataPreprocessor \
  --packages com.vdurmont:emoji-java:5.1.1 \
  target/scala-2.12/tweet-sentiment_2.12-1.0.jar

Step 3 — Train the Emotion Model

Trains a One-vs-Rest LinearSVC pipeline on the cleaned emotion Parquet and saves the model to HDFS.

spark-submit emotion_svc.py

Step 4 — Preprocess Sentiment Data

Cleans the Sentiment140 CSV (~10 GB) on HDFS and writes Parquet output. Allocate generous memory.

spark-submit \
  --class TweetPreprocessor \
  --executor-memory 4G \
  --driver-memory 4G \
  target/scala-2.12/tweet-sentiment_2.12-1.0.jar

Step 5 — Train the Sentiment Model

Trains a Logistic Regression pipeline (TF-IDF + bigrams) on the cleaned sentiment Parquet and saves the model to HDFS.

spark-submit \
  --master spark://127.0.0.1:7077 \
  --executor-memory 4G \
  --driver-memory 4G \
  --conf spark.sql.shuffle.partitions=16 \
  sentiment_LR.py

Step 6 — Start Real-Time Streaming Inference

Loads both trained models from HDFS, listens to the Kafka topic, and writes predictions to MongoDB. Keep this running while using the dashboard.

spark-submit \
  --master spark://127.0.0.1:7077 \
  --executor-memory 4G \
  --driver-memory 4G \
  --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.5.1 \
  streaming_dual_prediction.py

Step 7 — Launch the Dashboard

streamlit run dashboard.py

The dashboard auto-refreshes every 3 seconds, displays live sentiment and emotion distributions from the last 200 MongoDB predictions, and provides a text input widget that publishes messages directly to the Kafka topic.

ML Models

Sentiment Model — Logistic Regression

Parameter Value
Feature representation TF-IDF (unigrams, 8K dims) + bigram HashingTF (8K dims) = 16K dims
Minimum document frequency 5
Regularisation (λ) 0.001 (L2)
Max iterations 30
Optimiser L-BFGS
Train / test split 80 / 20 stratified

Evaluation (20% test set):

Metric Score
Accuracy 77.44%
Weighted Precision 77.51%
Weighted Recall 77.44%
Weighted F1-Score 77.40%

Emotion Model — One-vs-Rest LinearSVC

Parameter Value
Feature representation Bigram TF-IDF (20K dims)
Regularisation (λ) 0.1
Max iterations 50
Classes 6 (sadness, joy, love, anger, fear, surprise)
Train / test split 80 / 20 stratified

Evaluation (20% test set):

Metric Score
Accuracy 79.82%
Weighted F1-Score 79.47%

HDFS Paths

Data Path
Raw sentiment CSV hdfs://localhost:9000/tweetanalysis/tweets_10GB.csv
Raw emotion CSV hdfs://localhost:9000/tweetanalysis/emotion_data.csv
Cleaned sentiment Parquet hdfs://localhost:9000/tweetanalysis/processed/cleaned_tweets
Cleaned emotion Parquet hdfs://localhost:9000/tweetanalysis/processed/cleaned_emotion_data
Sentiment model hdfs://localhost:9000/tweetanalysis/models/sentiment_lr
Emotion model hdfs://localhost:9000/tweetanalysis/models/emotion_lr

Kafka Message Format

Publish JSON messages to the tweetanalyse topic:

{"text": "I am very happy today!"}

The streaming job reads the text field, runs it through both preprocessing pipelines and models, and emits:

{
  "text": "I am very happy today!",
  "sentiment_label": "positive",
  "emotion_label": "joy"
}

MongoDB Schema

Collection: tweetanalysis.predictions

{
  "_id": "<ObjectId>",
  "text": "<original tweet text>",
  "sentiment_label": "positive | negative",
  "emotion_label": "sadness | joy | love | anger | fear | surprise"
}

Pre-flight Checklist

Before starting the streaming job or dashboard, verify:

  • HDFS is running and raw CSV files exist at the configured paths
  • Spark standalone master is reachable at spark://localhost:7077
  • Both trained models are present in HDFS (models/sentiment_lr, models/emotion_lr)
  • Kafka broker is running and topic tweetanalyse exists
  • MongoDB is running and accessible at localhost:27017
  • Python dependencies (pyspark, pymongo, streamlit, plotly, kafka-python) are installed