Skip to content

perf(data_loader): cache JSON data in memory to avoid repeated disk reads - #13

Merged
komalharshita merged 1 commit into
komalharshita:mainfrom
Alm0stSurely:perf/cache-json-loading
May 7, 2026
Merged

perf(data_loader): cache JSON data in memory to avoid repeated disk reads#13
komalharshita merged 1 commit into
komalharshita:mainfrom
Alm0stSurely:perf/cache-json-loading

Conversation

@Alm0stSurely

Copy link
Copy Markdown
Contributor

Problem

As described in #11, utils/data_loader.py currently reads projects.json from disk on every call to load_all_projects().

Solution

Implemented in-memory cache with module-level variable:

  • _projects_cache stores data after first read
  • clear_cache() for test isolation
  • Updated tests to reset cache in setup_module()

Benchmarks

  • Cached: 0.06 µs per call
  • Uncached: 51.84 µs per call
  • Speedup: 847x

Testing

All 25 tests pass.

Fixes #11

…eads

Implements a simple module-level cache for projects.json to eliminate
repeated disk I/O on every call to load_all_projects().

Changes:
- Add _projects_cache module variable
- Modify load_all_projects() to use cache after first read
- Add clear_cache() function for test isolation
- Update tests to call clear_cache() in setup_module()
- Add benchmark script showing 847x speedup

Fixes komalharshita#11

@komalharshita komalharshita left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a solid improvement overall.

The caching logic is simple, clean, and fits the project really well. I also like that you added clear_cache() for test isolation and included benchmark results to show the performance improvement. The code stays beginner-friendly while still improving efficiency, which is exactly the direction we want for DevPath.

I reviewed the updated files and everything looks good to merge.

@komalharshita
komalharshita merged commit 1417ad7 into komalharshita:main May 7, 2026
1 check passed
@komalharshita

Copy link
Copy Markdown
Owner

Nice work on this optimization.
Reading the JSON file on every request was unnecessary, and the in-memory cache solves that cleanly without adding extra complexity. The benchmark script and test updates were a great addition too.
Thanks for keeping the implementation readable and well-documented.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Optimise data loading to avoid reading the JSON file on every request

2 participants