📊 Market Intelligence Platform

Production-Grade Marketplace Data Engineering & Analytics

A production-style Data Engineering platform that ingests eBay marketplace data (via official APIs), builds a metadata-driven Medallion Architecture, and delivers business-ready analytical datasets and AI-ready marts.

Objective • Tech Stack • Architecture • Ingestion Strategy • Current Focus • Structure • Status & Roadmap

---

🎯 Objective

The goal of this project is to demonstrate modern Data Engineering best practices by building an end-to-end data platform, including:

• 🔄 API Ingestion: Reliable extraction from high-volume marketplace REST APIs.
• ⚡ Incremental Loading: State-aware, cursor-based pagination and data delta processing.
• ☁️ Google Cloud Storage (GCS): Scalable cloud data lake storage for raw and curated data.
• ⚙️ Metadata-Driven Pipelines: Dynamic configuration-driven orchestration and schema management.
• 🥉🥈🥇 Medallion Architecture: Multi-hop Bronze, Silver, and Gold data transformations.
• 🚀 PySpark & Databricks: Distributed large-scale data processing and transformation.
• 📈 Business Data Marts: Curated analytical datasets for reporting and business intelligence.
• 🤖 Future RAG / BI Interface: Natural language querying and intelligence interface for decision makers.

---

🛠️ Technology Stack

Category	Technology	Role in Architecture
Cloud	Google Cloud Platform (GCP)	Cloud infrastructure and enterprise IAM security
Storage	Google Cloud Storage (GCS)	Raw landing bucket and intermediate data lake storage
Ingestion	DLTHub	REST API connectivity, state management, and retry handling
Processing	PySpark	Large-scale data transformations, cleansing, and aggregations
Platform	Databricks	Managed Spark runtime and lakehouse orchestration
Language	Python	Unified language across ingestion, pipelines, and tooling
Version Control	Git	Source code management and version control

---

🏗️ High-Level Architecture

The platform processes marketplace data through a multi-tier pipeline:

┌─────────────────────────────────────────────────────────────┐
│               Official eBay API (Browse API)                │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│                  DLTHub (Ingestion Layer)                   │
│    • API Connectivity  • Authentication  • Pagination       │
│    • Incremental State • Retries         • Concurrency      │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│                      GCS (Raw Bucket)                       │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│              Metadata-Driven PySpark Pipeline               │
└──────┬───────────────────────┼───────────────────────┬──────┘
       │                       │                       │
       ▼                       ▼                       ▼
 🥉 [ Bronze ]           🥈 [ Silver ]           🥇 [ Gold ]
  Raw Ingestion           Cleansed & Conformed    Analytical Datasets
  Delta / Parquet         Business Logic          & Data Marts
       │                       │                       │
       └───────────────────────┼───────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│          Business-Ready Analytical Data & RAG / BI          │
└─────────────────────────────────────────────────────────────┘

---

📥 Ingestion Strategy

The ingestion layer is being designed around the capabilities and constraints of the eBay Browse API. The current approach separates discovery from enrichment:

┌───────────────────────────────┐
│    eBay Search / Discovery    │  ──► Broad marketplace discovery
└──────────────┬────────────────┘
               │
               ▼
┌───────────────────────────────┐
│            itemId             │  ──► Harvested unique item identifiers
└──────────────┬────────────────┘
               │
               ▼
┌───────────────────────────────┐
│     Item-Level Enrichment     │  ──► Deep item-level product data & specs
└──────────────┬────────────────┘
               │
               ▼
┌───────────────────────────────┐
│     Raw Marketplace Data      │  ──► Landing in Google Cloud Storage (GCS)
└───────────────────────────────┘

Why this design?
The Search API provides broad marketplace discovery, while item-level endpoints can provide substantially richer product information.

Division of Responsibilities

 ┌──────────────────────────────────────┐     ┌──────────────────────────────────────┐
 │         DLTHub Ingestion             │     │        PySpark Transformation        │
 ├──────────────────────────────────────┤     ├──────────────────────────────────────┤
 │  • API connectivity                  │     │  • Bronze layer transformation       │
 │  • Authentication                    │     │  • Silver layer data cleansing       │
 │  • Pagination                        │ ──► │  • Gold layer analytical datasets    │
 │  • Incremental state                 │     │  • Business data mart construction   │
 │  • Retries                           │     │                                      │
 │  • Raw ingestion                     │     │                                      │
 │  • Request execution & concurrency   │     │                                      │
 └──────────────────────────────────────┘     └──────────────────────────────────────┘

• DLTHub is responsible for:

  • API connectivity
  • Authentication
  • Pagination
  • Incremental state
  • Retries
  • Raw ingestion
  • Request execution and concurrency

• The downstream PySpark layer will remain responsible for:

  • Bronze, Silver, and Gold transformations

---

🔍 Current Development Focus

The current development phase is focused on building and validating the DLTHub ingestion layer.

📌 Work completed so far includes:

• eBay Browse API source assessment
• eBay API authentication setup
• GCP project and service account configuration
• GCS raw and curated storage setup
• Initial DLTHub REST API ingestion implementation
• Custom request logging for API observability
• Investigation of DLTHub REST client and request execution
• Evaluation of DLT parallel extraction for independent API requests

---

📁 Project Structure

Market-Intelligence-Platform/
│
├── docs/                     # Documentation and architecture designs
├── infrastructure/           # Cloud infrastructure and IAM setup
├── ingestion-dlt/            # DLTHub ingestion project and API extraction
│   ├── config/               # API & category configuration YAMLs
│   ├── pipelines/            # Ingestion pipeline definitions
│   ├── sources/              # DLT REST API source logic
│   └── utils/                # Logging, date windowing & helper functions
└── spark-pipeline/           # Metadata-driven PySpark & Databricks pipelines

---

📊 Current Status & Roadmap

🚦 Current Status

• ✅ Architecture finalized
• ✅ Source assessment completed
• ✅ GCP environment configured
• ✅ GCS buckets created
• ✅ eBay API authentication established
• ✅ DLTHub ingestion project initialized
• ✅ DLTHub REST API implementation investigated
• 🚧 DLT ingestion pipeline under development
• 🚧 DLT parallel request execution being validated
• ⏳ Raw ingestion validation
• ⏳ Bronze layer
• ⏳ Silver layer
• ⏳ Gold analytical datasets
• ⏳ Business intelligence / RAG interface

---

🗺️ Roadmap

Phase 1 — Source & Ingestion

• eBay API integration
• Incremental ingestion
• Pagination and retry handling
• Parallel request execution
• Raw data storage in GCS

Phase 2 — Medallion Architecture

• Bronze layer
• Silver transformations
• Data quality checks
• Gold analytical datasets

Phase 3 — Analytics

• Marketplace trends
• Category analysis
• Product and seller insights
• Business-oriented data marts

Phase 4 — Intelligence Interface

• Natural-language business queries
• Analytical / RAG capabilities
• Interfaces for product managers, brand managers, and category managers

---

_{Built with modern Data Engineering principles. Maintained for production-readiness & scalable analytics.}