Tweet Sentiment & Emotion Analysis — Scalable Big Data Pipeline

A distributed, real-time tweet analytics system that simultaneously performs binary sentiment classification (positive / negative) and six-class emotion recognition (sadness, joy, love, anger, fear, surprise) on large-scale Twitter data. The pipeline is end-to-end: raw CSVs on HDFS → Spark batch preprocessing → MLlib model training → live Kafka stream inference → MongoDB persistence → Streamlit dashboard.

---

Table of Contents

• Overview
• Architecture
• Tech Stack
• Repository Structure
• Datasets
• Prerequisites
• Setup & Configuration
• Running the Pipeline
  • Step 1 — Build the Scala JAR
  • Step 2 — Preprocess Emotion Data
  • Step 3 — Train the Emotion Model
  • Step 4 — Preprocess Sentiment Data
  • Step 5 — Train the Sentiment Model
  • Step 6 — Start Real-Time Streaming Inference
  • Step 7 — Launch the Dashboard
• ML Models
• HDFS Paths
• Kafka Message Format
• MongoDB Schema
• Pre-flight Checklist

---

Overview

The system is designed to process tweet-scale data (tens of millions of records) using a fully distributed stack. Batch preprocessing and model training are handled by Apache Spark jobs running on a Spark standalone cluster backed by HDFS. Trained models are serialised as Spark MLlib PipelineModel objects and stored on HDFS. At inference time, Apache Spark Structured Streaming consumes a Kafka topic as micro-batches, applies both models concurrently, and writes dual-labelled predictions to MongoDB. A Streamlit dashboard reads from MongoDB in real time to display live results and allows analysts to inject arbitrary tweet text via an embedded Kafka producer.

---

Architecture

Raw CSVs (HDFS)
      │
      ▼
┌─────────────────────────────────────────────┐
│  Spark Batch Preprocessing  (Scala / SBT)   │
│  • URL / mention / hashtag removal          │
│  • Emoji transliteration (emoji-java)       │
│  • Lowercasing, punctuation stripping       │
│  → Parquet output back to HDFS              │
└─────────────────────────────────────────────┘
      │
      ▼
┌─────────────────────────────────────────────┐
│  Spark MLlib Model Training  (PySpark)      │
│  • Sentiment: TF-IDF + Bigrams → LR        │
│  • Emotion:   Bigram TF-IDF  → OvR + SVC   │
│  → PipelineModel serialised to HDFS         │
└─────────────────────────────────────────────┘
      │
      ▼
┌─────────────────────────────────────────────┐
│  Apache Kafka  (topic: tweetanalyse)        │
│  JSON messages: {"text": "..."}             │
└─────────────────────────────────────────────┘
      │
      ▼
┌─────────────────────────────────────────────┐
│  Spark Structured Streaming Inference       │
│  • Loads both PipelineModels from HDFS      │
│  • Applies sentiment + emotion models       │
│  • Merges dual labels per record            │
│  → Writes to MongoDB (foreachBatch)         │
└─────────────────────────────────────────────┘
      │
      ▼
┌─────────────────────────────────────────────┐
│  Streamlit Dashboard                        │
│  • Live sentiment / emotion charts          │
│  • Embedded Kafka producer (tweet input)    │
└─────────────────────────────────────────────┘

---

Tech Stack

Component	Technology
Distributed compute	Apache Spark 3.5.1 (standalone cluster)
Batch preprocessing	Scala 2.12.18, SBT, emoji-java 5.1.1
ML training & inference	PySpark MLlib (Logistic Regression, LinearSVC, OvR)
Distributed storage	Hadoop HDFS 3.x (NameNode localhost:9000)
Message broker	Apache Kafka (broker localhost:9092)
Database	MongoDB (localhost:27017)
Dashboard	Streamlit + Plotly

---

Repository Structure

.
├── src/
│   └── main/scala/
│       ├── TweetPreprocessor.scala        # Spark batch job — sentiment data cleaning
│       └── EmotionDataPreprocessor.scala  # Spark batch job — emotion data cleaning
├── sentiment_LR.py                        # PySpark — train Logistic Regression (sentiment)
├── emotion_svc.py                         # PySpark — train OvR LinearSVC (emotion)
├── streaming_dual_prediction.py           # PySpark Structured Streaming — live inference
├── dashboard.py                           # Streamlit dashboard
├── build.sbt                              # SBT build definition
├── run.txt                                # Quick-reference run commands
└── README.md

---

Datasets

Dataset	Records	Classes	Description
Sentiment140	~13.5 M	2 (pos / neg)	Tweets with distant-supervision binary labels
CARER Emotion	~20 K	6 (sadness, joy, love, anger, fear, surprise)	Fine-grained emotion text dataset

Place raw CSV files on HDFS before running the pipeline (see HDFS Paths).

---

Prerequisites

Ensure the following are installed and running before executing any pipeline step:

• Java 8 or 11 (compatible with Spark 3.5.x)
• Scala 2.12 + SBT
• Apache Spark 3.5.1 — standalone master expected at spark://localhost:7077
• Hadoop HDFS — NameNode expected at hdfs://localhost:9000
• Python 3.8+ with the packages below:

  pyspark==3.5.1
  pymongo
  streamlit
  plotly
  kafka-python
  

• Apache Kafka — broker expected at localhost:9092; create topic before streaming:

  kafka-topics.sh --create --topic tweetanalyse --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

• MongoDB — expected at mongodb://127.0.0.1:27017

---

Setup & Configuration

All service endpoints are hard-coded in the respective scripts. If your environment differs, update the following:

Setting	Default	File(s)
Spark master	spark://localhost:7077	sentiment_LR.py, emotion_svc.py, streaming_dual_prediction.py
HDFS NameNode	hdfs://localhost:9000	All Scala & Python files
Kafka broker	localhost:9092	streaming_dual_prediction.py, dashboard.py
Kafka topic	tweetanalyse	streaming_dual_prediction.py, dashboard.py
MongoDB URI	mongodb://127.0.0.1:27017	streaming_dual_prediction.py, dashboard.py
MongoDB DB / collection	tweetanalysis / predictions	streaming_dual_prediction.py, dashboard.py

---

Running the Pipeline

Execute the steps in order. Steps 1–5 are offline (batch); steps 6–7 are online (streaming).

Step 1 — Build the Scala JAR

sbt package

Output: target/scala-2.12/tweet-sentiment_2.12-1.0.jar

---

Step 2 — Preprocess Emotion Data

Cleans the CARER emotion CSV on HDFS and writes Parquet output.

spark-submit \
  --class EmotionDataPreprocessor \
  --packages com.vdurmont:emoji-java:5.1.1 \
  target/scala-2.12/tweet-sentiment_2.12-1.0.jar

---

Step 3 — Train the Emotion Model

Trains a One-vs-Rest LinearSVC pipeline on the cleaned emotion Parquet and saves the model to HDFS.

spark-submit emotion_svc.py

---

Step 4 — Preprocess Sentiment Data

Cleans the Sentiment140 CSV (~10 GB) on HDFS and writes Parquet output. Allocate generous memory.

spark-submit \
  --class TweetPreprocessor \
  --executor-memory 4G \
  --driver-memory 4G \
  target/scala-2.12/tweet-sentiment_2.12-1.0.jar

---

Step 5 — Train the Sentiment Model

Trains a Logistic Regression pipeline (TF-IDF + bigrams) on the cleaned sentiment Parquet and saves the model to HDFS.

spark-submit \
  --master spark://127.0.0.1:7077 \
  --executor-memory 4G \
  --driver-memory 4G \
  --conf spark.sql.shuffle.partitions=16 \
  sentiment_LR.py

---

Step 6 — Start Real-Time Streaming Inference

Loads both trained models from HDFS, listens to the Kafka topic, and writes predictions to MongoDB. Keep this running while using the dashboard.

spark-submit \
  --master spark://127.0.0.1:7077 \
  --executor-memory 4G \
  --driver-memory 4G \
  --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.5.1 \
  streaming_dual_prediction.py

---

Step 7 — Launch the Dashboard

streamlit run dashboard.py

The dashboard auto-refreshes every 3 seconds, displays live sentiment and emotion distributions from the last 200 MongoDB predictions, and provides a text input widget that publishes messages directly to the Kafka topic.

---

ML Models

Sentiment Model — Logistic Regression

Parameter	Value
Feature representation	TF-IDF (unigrams, 8K dims) + bigram HashingTF (8K dims) = 16K dims
Minimum document frequency	5
Regularisation (λ)	0.001 (L2)
Max iterations	30
Optimiser	L-BFGS
Train / test split	80 / 20 stratified

Evaluation (20% test set):

Metric	Score
Accuracy	77.44%
Weighted Precision	77.51%
Weighted Recall	77.44%
Weighted F1-Score	77.40%

---

Emotion Model — One-vs-Rest LinearSVC

Parameter	Value
Feature representation	Bigram TF-IDF (20K dims)
Regularisation (λ)	0.1
Max iterations	50
Classes	6 (sadness, joy, love, anger, fear, surprise)
Train / test split	80 / 20 stratified

Evaluation (20% test set):

Metric	Score
Accuracy	79.82%
Weighted F1-Score	79.47%

---

HDFS Paths

Data	Path
Raw sentiment CSV	hdfs://localhost:9000/tweetanalysis/tweets_10GB.csv
Raw emotion CSV	hdfs://localhost:9000/tweetanalysis/emotion_data.csv
Cleaned sentiment Parquet	hdfs://localhost:9000/tweetanalysis/processed/cleaned_tweets
Cleaned emotion Parquet	hdfs://localhost:9000/tweetanalysis/processed/cleaned_emotion_data
Sentiment model	hdfs://localhost:9000/tweetanalysis/models/sentiment_lr
Emotion model	hdfs://localhost:9000/tweetanalysis/models/emotion_lr

---

Kafka Message Format

Publish JSON messages to the tweetanalyse topic:

{"text": "I am very happy today!"}

The streaming job reads the text field, runs it through both preprocessing pipelines and models, and emits:

{
  "text": "I am very happy today!",
  "sentiment_label": "positive",
  "emotion_label": "joy"
}

---

MongoDB Schema

Collection: tweetanalysis.predictions

{
  "_id": "<ObjectId>",
  "text": "<original tweet text>",
  "sentiment_label": "positive | negative",
  "emotion_label": "sadness | joy | love | anger | fear | surprise"
}

---

Pre-flight Checklist

Before starting the streaming job or dashboard, verify:

• HDFS is running and raw CSV files exist at the configured paths
• Spark standalone master is reachable at spark://localhost:7077
• Both trained models are present in HDFS (models/sentiment_lr, models/emotion_lr)
• Kafka broker is running and topic tweetanalyse exists
• MongoDB is running and accessible at localhost:27017
• Python dependencies (pyspark, pymongo, streamlit, plotly, kafka-python) are installed