Skip to content

Feat: Migrate Data Source from JSON to Relational Database (SQLAlchem…#1194

Open
desireddymohithreddy0925 wants to merge 3 commits into
komalharshita:mainfrom
desireddymohithreddy0925:feat/issue-1176-sqlalchemy-migration
Open

Feat: Migrate Data Source from JSON to Relational Database (SQLAlchem…#1194
desireddymohithreddy0925 wants to merge 3 commits into
komalharshita:mainfrom
desireddymohithreddy0925:feat/issue-1176-sqlalchemy-migration

Conversation

@desireddymohithreddy0925

Copy link
Copy Markdown

Summary

This PR resolves #1176 by migrating the DevPath data layer from a static projects.json file to an SQLite relational database via Flask-SQLAlchemy. This lays the groundwork necessary to implement dynamic user-specific features like progress tracking, bookmarks, and upvoting without risking race conditions or data loss in flat files.

Related Issue

Closes #1176

Type of Change

  • Bug fix — resolves a broken behaviour
  • Feature — adds new functionality
  • Data — adds new projects to data/projects.json
  • Documentation — updates docs, README, or code comments only
  • Style — CSS or visual changes only, no logic change
  • Refactor — restructures code without changing behaviour
  • Test — adds or updates tests

What Was Changed

File Change made
requirements.txt Added Flask-SQLAlchemy==3.1.1 as a core dependency.
src/config.py Defined the SQLALCHEMY_DATABASE_URI configuration pointing to data/devpath.db.
src/models.py Added a new Project SQLAlchemy model mapped to the projects table. Utilized db.JSON for arrays (skills, features, roadmap, etc.) to match our existing data structure, alongside a .to_dict() helper method.
src/app.py Initialized the SQLAlchemy db object with the main Flask app context.
src/seed_db.py Authored a one-time migration script. This iterates through projects.json and cleanly populates the new SQLite database.
src/utils/data_loader.py Removed all JSON file I/O operations and module-level static caching. Refactored load_all_projects and find_project_by_id to directly query the database and serialize the results back into standard Python dictionaries, keeping the machine learning recommender completely intact.
tests/test_basic.py Updated setup_module() to inject an in-memory SQLite database (sqlite:///<tmp>) within the test client's app context and populate it with our existing JSON file data before executing any of the 80 tests.

How to Test This PR

  1. Checkout the branch: git checkout feat/issue-1176-sqlalchemy-migration
  2. Run pip install -r requirements.txt to grab the SQLAlchemy package.
  3. Run python src/seed_db.py to perform the initial migration and generate data/devpath.db.
  4. Run pytest tests/test_basic.py and ensure all 80 tests pass without failures.
  5. Boot up the app: python src/app.py
  6. Fill out the frontend form to verify the backend successfully executes queries and generates recommendations from the new database architecture.

Test Results

============================= test session starts ==============================
platform darwin -- Python 3.9.6, pytest-8.2.2, pluggy-1.6.0
rootdir: /Users/desireddymohithreddy/GSSOC2026DevPath/DevPathMRD
collected 80 items

tests/test_basic.py .................................................... [ 65%]
............................                                             [100%]

=============================== warnings summary ===============================
tests/test_basic.py::test_find_project_by_id_found
...
  /Users/desireddymohithreddy/GSSOC2026DevPath/DevPathMRD/src/utils/data_loader.py:84: LegacyAPIWarning: The Query.get() method is considered legacy as of the 1.x series of SQLAlchemy and becomes a legacy construct in 2.0. The method is now available as Session.get() (deprecated since: 2.0) (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
    p = Project.query.get(project_id)

======================== 80 passed, 9 warnings in 1.10s ========================

Self-Review Checklist

  • I have read CONTRIBUTING.md and followed all guidelines
  • My branch name follows the convention: feat/, fix/, docs/, data/, style/, test/
  • I have run pytest tests/test_basic.py and all tests pass
  • I have run flake8 . locally and there are no errors
  • I have not introduced any print() or console.log() debug statements
  • Every new function I wrote has a docstring
  • I have not modified files outside the scope of the linked issue
  • If I changed the UI, I tested it at 375px (mobile) and 1280px (desktop)
  • If I added a project to the dataset, it has all required JSON fields

Notes for Reviewer

None.

…y) (komalharshita#1176)

- Added Flask-SQLAlchemy to requirements
- Created Project db.Model with db.JSON support
- Wrote seed_db.py script to migrate JSON to SQLite
- Refactored load_all_projects to query SQLite db
- Updated test_basic.py to use temporary in-memory sqlite db
@vercel

vercel Bot commented Jun 26, 2026

Copy link
Copy Markdown

Someone is attempting to deploy a commit to the komalsony234-1530's projects Team on Vercel.

A member of the Team first needs to authorize it.

@komalharshita

Copy link
Copy Markdown
Owner

@desireddymohithreddy0925 kindly resolve the merge conflicts

@desireddymohithreddy0925

Copy link
Copy Markdown
Author

@komalharshita done mam

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Migrate data source from projects.json to a relational database (SQLAlchemy)

2 participants