Serverless Data Lakehouse with Apache Iceberg

A fully containerized Serverless Lakehouse architecture demonstrating Medallion architecture (Bronze, Silver, Gold), ACID transactions on a data lake using Apache Iceberg, and PySpark for big data processing.

Tech Stack

• Storage: MinIO (AWS S3 Compatible Object Storage)
• Compute: PySpark / Jupyter
• Table Format: Apache Iceberg
• Infrastructure: Docker & Docker Compose

Architecture Flow

This project implements the Medallion Architecture:

1. Bronze Layer: Raw yellow_tripdata Parquet files ingested directly into MinIO storage as Apache Iceberg tables.
2. Silver Layer: Filtered and cleaned data. Demonstrated Iceberg's ACID compliance by performing row-level UPDATE aggregations and ALTER TABLE schema evolutions directly on the S3 bucket.
3. Gold Layer: Heavily aggregated business-level tables (e.g., daily revenue, ride counts) structured specifically for fast BI and visualization querying.

How to Run

1. Ensure Docker and Docker Compose are installed.
2. Run the infrastructure:

   docker-compose up -d

3. Log into the MinIO console (localhost:9001) with admin/password and create a bucket named lakehouse.
4. Open the Jupyter notebook via the Docker logs and run all cells sequentially to trigger the PySpark ETL process and plot the final Gold datasets.