|
1 | | -# IBEX-Scraper-API |
| 1 | +# IBEX-Scraper-API |
| 2 | + |
| 3 | +A secure ASP.NET Core Web API that automatically scrapes **day-ahead prices from the Free Electricity Market (Day Ahead segment) on IBEX** and stores them in a structured Azure SQL database for use by external applications. |
| 4 | + |
| 5 | +## ⚡ Purpose |
| 6 | + |
| 7 | +This API was built to **decouple IBEX access from dependent systems** by providing a secure, reliable, and filterable data backend. |
| 8 | + |
| 9 | +It powers apps like [SunNext](https://github.com/JacksonJS12/SunNext) by allowing them to: |
| 10 | +- Access daily price data without scraping IBEX directly |
| 11 | +- Query historical electricity prices by date/hour |
| 12 | +- Build custom dashboards and energy trading strategies |
| 13 | + |
| 14 | +## 📊 What It Does |
| 15 | + |
| 16 | +- Scrapes **IBEX Day Ahead segment** prices for the next day |
| 17 | +- Extracts hourly energy prices (EUR/MWh) |
| 18 | +- Saves results into an Azure SQL `MarketPrices` table with: |
| 19 | + - `Date` (e.g. `day/month/year`) |
| 20 | + - `Hour` (1–24 using Central European Time – CET Time Zone) |
| 21 | + - `PricePerMWh` (BGN/MWh) |
| 22 | + |
| 23 | +## 🕒 Scheduling |
| 24 | + |
| 25 | +- Powered by an **Azure Logic App** that triggers daily at **14:00 BG time (UTC+3)** |
| 26 | +- Calls the `POST /api/Scrape/scrape-and-save` endpoint |
| 27 | +- Only stores new data (prevents duplicate entries for the same day) |
| 28 | + |
| 29 | +## 🧱 Tech Stack |
| 30 | + |
| 31 | +- **Backend:** ASP.NET Core Web API (.NET 8) |
| 32 | +- **Scraper:** AngleSharp |
| 33 | +- **Database:** Azure SQL |
| 34 | +- **Scheduler:** Azure Logic Apps |
| 35 | +- **Deployment:** Azure App Service |
| 36 | +- **CI/CD:** GitHub Actions |
| 37 | + |
| 38 | +📁 Project Structure |
| 39 | +IBEX-Scraper-API/ <br/> |
| 40 | +├── Controllers/ <br/> |
| 41 | +│ └── ScrapeController.cs <br/> |
| 42 | +├── Services/ <br/> |
| 43 | +│ └── IbexScraper/ <br/> |
| 44 | +│ └── IbexScraperService.cs <br/> |
| 45 | +│ └── IIbexScraperService.cs <br/> |
| 46 | +├── Data/ <br/> |
| 47 | +│ └── Migrations/ <br/> |
| 48 | +│ └── Models/ <br/> |
| 49 | +│ └── MarketPrice.cs <br/> |
| 50 | +│ └── AppDbContext.cs <br/> |
| 51 | +│ └── AppDbContextFactory.cs <br/> |
| 52 | +└── Program.cs <br/> |
| 53 | + |
0 commit comments