feat(websoc-scraper): Dynamic Resolution for Enrollment History#300
Open
aadi-shanker wants to merge 23 commits intomainfrom
Open
feat(websoc-scraper): Dynamic Resolution for Enrollment History#300aadi-shanker wants to merge 23 commits intomainfrom
aadi-shanker wants to merge 23 commits intomainfrom
Conversation
…date createdAt to timestamp
…ollment management
…ng in websoc scraper
ParzivalPerhaps
approved these changes
Feb 16, 2026
Contributor
ParzivalPerhaps
left a comment
There was a problem hiding this comment.
Aadi owes me $10,000
lgtm
laggycomputer
requested changes
Feb 16, 2026
HwijungK
reviewed
Feb 16, 2026
HwijungK
reviewed
Feb 16, 2026
laggycomputer
requested changes
Feb 19, 2026
laggycomputer
requested changes
Feb 19, 2026
Member
There was a problem hiding this comment.
This is fine for now, but there are a lot of small constants which will eventually live here.
5 tasks
laggycomputer
requested changes
Feb 19, 2026
Member
|
Good to go, but I will not approve so that merging is blocked while we do our deployment to a separate worker. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Implements variable-frequency enrollment snapshots based on academic calendar periods. The scraper now captures enrollment data at different frequencies depending on enrollment activity:
ENROLLMENT period (Week 8-10): Every 3 hours, 7am-7pm only (~5 snapshots/day)
ADD_DROP period (Week 1-2): Every 6 hours 24/7, with hourly snapshots on Week 2 Friday 12pm-5pm (~4-8 snapshots/day)
REGULAR period (Week 3-7): Once per week (168 hours)
Between quarters: Once per week for late enrollment
Database-driven frequency tracking checks hours elapsed since the last snapshot and only inserts when thresholds are met. This prevents missed snapshots during scraper outages (self-correcting behavior) while optimizing storage usage.
Related Issue
#134
Motivation and Context
Missing intraday enrollment trends -Want to increase and visualize hourly changes during critical periods
How Has This Been Tested?
I ran a test file with multiple detection unit tests based on given fake dates. I also ran the scraper using fake dates just to double check the processing logic is correct.
Screenshots (if appropriate):
Types of changes
Checklist: