Skip to content

feat(websoc-scraper): Dynamic Resolution for Enrollment History#300

Open
aadi-shanker wants to merge 23 commits intomainfrom
enrollmentdatafix
Open

feat(websoc-scraper): Dynamic Resolution for Enrollment History#300
aadi-shanker wants to merge 23 commits intomainfrom
enrollmentdatafix

Conversation

@aadi-shanker
Copy link
Contributor

@aadi-shanker aadi-shanker commented Feb 5, 2026

Description

Implements variable-frequency enrollment snapshots based on academic calendar periods. The scraper now captures enrollment data at different frequencies depending on enrollment activity:

ENROLLMENT period (Week 8-10): Every 3 hours, 7am-7pm only (~5 snapshots/day)
ADD_DROP period (Week 1-2): Every 6 hours 24/7, with hourly snapshots on Week 2 Friday 12pm-5pm (~4-8 snapshots/day)
REGULAR period (Week 3-7): Once per week (168 hours)
Between quarters: Once per week for late enrollment

Database-driven frequency tracking checks hours elapsed since the last snapshot and only inserts when thresholds are met. This prevents missed snapshots during scraper outages (self-correcting behavior) while optimizing storage usage.

Related Issue

#134

Motivation and Context

Missing intraday enrollment trends -Want to increase and visualize hourly changes during critical periods

How Has This Been Tested?

I ran a test file with multiple detection unit tests based on given fake dates. I also ran the scraper using fake dates just to double check the processing logic is correct.

Screenshots (if appropriate):

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code involves a change to the database schema.
  • My code requires a change to the documentation.

@aadi-shanker aadi-shanker marked this pull request as ready for review February 13, 2026 01:53
laggycomputer

This comment was marked as resolved.

Copy link
Contributor

@ParzivalPerhaps ParzivalPerhaps left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aadi owes me $10,000

lgtm

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fine for now, but there are a lot of small constants which will eventually live here.

laggycomputer

This comment was marked as resolved.

@laggycomputer
Copy link
Member

Good to go, but I will not approve so that merging is blocked while we do our deployment to a separate worker.

@laggycomputer laggycomputer linked an issue Mar 7, 2026 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RFC: Higher Resolution for Enrollment History

4 participants